You are not logged in.
OK, thanks.
I prefer ab's original suggestion. For the most used datatype beside integer being short is an advantage.
If mORMot2 uses UTF8String:
I am wondering if using some short alias in my own code like UTF8 = UTF8String or U8 = UTF8String would cause any problem?
mORMot 2 - got it.
SHA-3 - sounds interesting.
Accept-Encoding , TLS, nginx - this project is not just about the typical CS scenario. I need extremely secure peer to peer communication in mixed environment with mobiles as well. I would rather not continue this subject publicly. You can answer my email if you have the time.
One more question:
6. Would it be possible to create an option to make aes encoding the default so it would work without declaring "ACCEPT-ENCODING: synshaaes" in the header? It would allow hiding which specific implementation of encryption is used from spectators.
Thanks for the very quick answer.
1)a) 11.6.5.3. in the documentation mentions AES256-CFB.
2) "SHA-3 in encryption mode " I do not quite get this. In my understanding SHA-3 is a hasher. Do you suggest to use it as an additional check for proper delivery? Or are you referring to JWT?
Got it, thanks.
I have just began digging in to the subject. A few questions:
1. Which is currently the strongest encryption in mORMot if speed is not that important
a) for messaging?
b) for database?
c) Is there a minimal data size to be effective?
2. Any other suggestions for very strong encryption beyond mORMot which is worth the extra effort?
3. Any future plans for mORMot?
4. Is the CTR mode mentioned here is the same as AES-256-CTR in mORMot?
https://www.highgo.ca/2019/08/08/the-di … algorithm/
5. Is the way mORMot tries to defend against replay attacks sufficient protection against the mentioned weakness of AES-CFB?
Wow, even more exciting.
I can see in the source now the part which loads ICU for android.
Could not find TICULibrary yet... I assume it is still work in progress.
AB,
this sounds interesting. ![]()
Since android is not fully posix compatible and bionic is lacking in this deparment maybe this is worth a look:
https://www.crystax.net/en/android/ndk/10
It allows to target IOS, OS X, Windows, Linux ...
mORMot is aimed mainly at microservices on servers but I am also interested in an other use case: local database access for desktop and mobile apps.
When they cancelled he development of the former v4 branch it did not feel right. Even if it was fundamentally faulty in some ways it should have been redesigned. Trial and error is the only way to get to the goal.
Which extensions do you expect to be available?
I am not familiar with sqlite's source, but for now it seems to be decimal.c only .
About decimal support, what if we added some curr_add() cur_sub() and cur_mul() functions, not working with arbitrary decimal length as decimal.c (which makes it somewhat slow), but using internally a pascal currency storage (i.e. 64-bit with 4 fixed decimals), and working from any kind of input: integer, double, and text.
It would be definitely an improvement on mORMots currency support to have "operators" on sql level.
The big question for me are aggregate functions on the sql level.
#This belongs more to an SQLite forum, but it has relevance for mORMot development too.#
I do not know if such ideas have ever been discussed for SQLite development, but the best solution I can think of if SQLite had an interface to assign a callback function for every aggregate based on the actual name of the the datatype.
This would allow developers to tell the engine if a certain aggregate is valid for the given datatype, and define how it is done.
Taking the extensibility even further: an interface to define datatypes
- their relations and conversion to other datatypes
- setter&getter functions for storage
- comparison functions
...
I know this is no small job for the developers (probably v4 level), but the flexibility it provides would be a huge benefit for the usability of the engine.
It would make extending the supported datatypes by SQLite itself very easy as well.
It would allow mORMot to make it's currency implementation work like a native SQLite datatype.
Going down on this road even further probably many limitations on virtual tables could be solved. Even creating and updating VTs structure by SQL could be made possible if the developers can provide the necessary functionality via interfaces.
You may not have the time or desire to work on including this specific extension, but in more general it would be useful to have a simple way to recompile sqlite with all the mORMot goodies and selected extensions for ourselves.
Maybe fpcupdeluxe could be extended with this feature. I assume beside AB Alf is the best qualified for this sort of stuff.
Wow, you are really quick!
There may not be a dedicated Decimal data type for a long time. But with the Decimal extension the functionality is at least available, even though it requires a different syntax because it is still a text field. It is not optimal, mORMot's way mapping it to int64 is better, but the main advantage for me would be the possibility to use decimal_sum.
AB,
the new version of SQLite is out. There is no mention about the decimal support this time. Is it now included in the amalgamation?
3.33 has been included.
They didn't include any 3.33.# after-release patch this time!
Take an umbrella to scare away the rain. ![]()
An other important way to look at this is that instead of making a decision about which setup to use we can continuously test and develop for multiply engines and setups with the ability to change the live setup any time with no extra effort.
ab,
for a long time I had this view that the best way to catch problems with DB engines is to test them against their previous versions and against other engines in an automated fashion.
(Because of differences in the SQL dialects this is not an obvious task, but frameworks like AnyDac or mORMot are already dealing with these differences. So even if it is not always the case for now I am going to assume that this will not be a big problem. )
Data access frameworks usually have a layer communicating with the engines. Testing could be made very easy, if this layer could send every request to several engines. And the result sets are automatically compared to each other. (The comparison can be done in the background, it would not keep the user/tester waiting. See the details later.) This way any change in the behavior of a new version can be detected automatically. The same goes if an engine handles something differently from other engines.
These can be caught automatically:
- different rows or field values in result sets
- different ordering of the result set
- longer response times than expected
- issues with transaction handling (these can be especially difficult to spot normally)
During the whole test phase up to beta or public every tester could test multiply engines without any extra job on anyone's part.
----------
In order this to work it has to be ensured that all databases are at the same state before a new SQL command is started. This can be done two ways:
1. Wait until all the engines finished the previous SQL command. This would slow down the test system.
2. Something similar to event sourcing. DB events has a serialized unique ID. The execution of the events is queued and logged for every tested engine. Practically this ID tells us where every engine is at on our virtual timeline.
The logs produced like this can be very informative about every specific real life application. It can show where the bottlenecks are with different systems and give a clear indication which db engine/hardware/server deployment combo is better for the specific use case. Migrating to an other engine can be made safer as well, since all db activity is automatically compered and tested for both systems.
It makes total sense to be cautious with the fresh releases.
Hopefully the end goal is to create a proper decimal type, but this is a step to the right direction. It is very much needed. I have never understood why was this not a priority for a long time. AFAIK the since cancelled v4 branch had proper decimal support. I assume this made the developers avoid the subject for v3 even longer.
I have no idea how difficult it is to add the extra source file(s) for mORMot, but it would be a little surprising if the source was not synced.
v3.33 is out with decimal support.
Will this be compiled into mORMot's sqlite?
I have run into the same problem recently. As mentioned earlier in this topic RawUTF8 is the same as the FPC String so one would expect that assigning a string constant should work seamlessly. Unfortunately it does not, but his solved it for me :
aRawUTF8:= StringToUTF8(aConstantString);
I was lamenting about the effects of changing the definition of RawUTF8 for FPC like this: RawUTF8 = String;
It might mess up some parts of mORMot. Even if all issues solved the code created based on this definition would be FPC only.
To answer my question, rookie mistake :
Instead
aRawUTF8:= 'ű';
Should be
aRawUTF8:= StringToUTF8('ű');
I wrongly assumed automatic conversion for constant values.
Thanks. This seems kind of low level. Could not find anything readily usable in it yet. I will check it out later. Luckily I already have an almost workable solution which is good enough for now. The only problem left to solve the one I mentioned before with TSynAnsiConvert.Engine(1250).UTF8ToAnsi. Is there any other way for RawUTF8 to AnsiString conversion?
Have you looked at this: https://www.yunqa.de/delphi/products/converters/index
Thanks, this may be useful some day.
If I understand your code right using NormToUpperAnsi7 is a speed optimiziation which is not codepage aware. If SortIStr/SortStr had a version using a slower CP aware comparison instead it would gave mORMot sorting abilities in any locale. The comparsion could be based on the RTL as you suggested. Not sure though how Delphi RTL is different in this regard.
Advice: do not reinvent the wheel about Unicode.
When you see the ICU table size and complexity, it is a whole project.
Rather use the RTL functions.
You are right of course. Being short on time I was looking for the fastest solution and adapting something for this purpose I had to create anyway for an other logically similar purpose seemed like a good idea.
I gave up rushing and went into a deep dive into RTL and the world of unicode, unicode collations, ICU and the database it is based on. Also looked into how mORMot handles sorting.
It was useful I understand the subject better now but honestly this was a total blackout on my part:
The original problem was that I could not find locale aware comparison routines for UTF-8. So I started this topic hoping I had missed something.
But there is an extremely simple solution for this, though not optimal for speed: converting to code page aware ansistrings and comparing those with AnsiCompareText gives the proper result.
I thought that this would be the proper way to convert from RawUTF8, but it did not no work:
var aRawUTF8: RawUTF8;
aAnsiString: AnsiString;
begin
aRawUTF8:= 'ű';
aAnsiString:= TSynAnsiConvert.Engine(1250).UTF8ToAnsi(aRawUTF8);
>> aAnsiString = '?' at this point
end;
mpv,
thanks, but I need something closer to home.
Unfortunately the only pascal wrapper is over a decade old: http://www.crossgl.com/icu4pas. This is such an essential functionality for none English programs that it would make sense to me to revitalize this project as part of FPC/Lazarus.
For now - since there does not seem to be a viable option for me - I am heading to carve my own path and create a general purpose string compare solution which can be easily extended for different locales. I need it mostly for the client side/local servers so it will not be optimized for speed at first.
Hi,
I am looking for ways to localize sorting on all platforms - Android, IOS included.
Anyone has experience with http://site.icu-project.org ?
ab,
by looking at the repository comments it seems decimal arithmetic is finally coming.
Strangely it is described as an extension for doing decimal arithmetic on strings - so it seems there is no proper decimal data type. (Or in SQLite's world "decimal affinity".)
Currently it is built into the shell, but is an optional add-on for the library. It is not included in the amalgamation.
If this remains the case, will this be a part of the mORMot engine?
I need to figure out how to reference and link the .o files with Delphi.
I am not really finding anything useful about this by googling.
Now I understand where you are coming from. Sounds like a good solution. ![]()
I need to figure out how to reference and link the .o files with Delphi.
In theory having a local server makes sense in some cases. EG when more than one app is using it. Also the server has tags telling the OS to keep it in memory as much as possible. The UI app will be removed before the server. The result is that at least half of the app has a chance to survive in memory --> faster reload.
If there is fast and reliable way for the UI app to check if the server is running and to start it if not then it might even work. ![]()
You are right, I did not think of Linux, only the mobile platforms.
CE 10.4 detto.
You can link a dynamic library (.so) on iOS.
So you need to compile the source as static libray (.o).
But it is a possible path for sure.WebSockets issue on mobile has no energy problem, but it may have reconnection issues. It is more common to have the network disconnected on a mobile, when you switch from one GSM tower to another.
Also on server side, HTTPS will use less resource - especially with a nginx as reserve-proxy. Currently, our WebSockets server consumes one thread per connected client, whereas our HTTP server is able to use a thread pool.
So I guess using regular HTTPS may be a better option.
Good to hear. ![]()
The goal is to have a local server on the mobile device. HTTP or HTTPS is better for this?
I would assume that local connections should not be effected by switching GSM tower. Maybe the device going to sleep mode and coming back needs to be handled.
EMK, encryption an unified data handling are the main driving factor for me too.
Potentially being able to access the same server remotely can be a benefit for support as well.
Using Android Studio to create the service might be better option because with Delphi even the service without GUI seems to consume lots of memory.
Creating the IOS background app with Lazarus instead of Delphi might be an option as well.
If you do not have any licences under your own name you could use the community edition. But it does not work with paid licences on the same machine.
Neverless I am really interested how sponsoring would work beside providing the licence?
ab,
are there any plans for the foreseeable future to make mORMot usable for android and ios directly?
Hi,
this is the concept:
1. Compile a mORMot server with Lazarus into a dynamic library
2. Create an Android "remote service"/"IOS background app" with Delphi using this dynamic library to start/stop/manage the mORMot server
3. Access the server from a FMX app via websockets.
Is this a viable path?
Does using websockets use more energy than using intents to communicate?
BeRo1985,
As you are deep into this stuff this is an off topic audio processing related question - you might be able to point me to the right direction :
I need a relatively simple functionality to split and insert into compressed audio files at a given time with Delphi or FP. All platforms are relevant but the most important ones are Windows, Android and IOS. So far I am struggling to find a working solution for this.
This is quite surprising. The foundation of closs-platform development should be unified API's at the deepest possible level.
I was hoping to use the mORMmot sqlite engine because it has encryption and is always up to date. The FireDac embedded engine has its own encryption, but it seems to be updated only occasionally. With dynamic linking encryption is not possible at all.
Thanks for the quick answer.
ab,
seems it may be compatible with mORMot for all platforms. Is mobile development with the regular mORMot units is a viable option now?
Faced with the same issues I came to conclusion that on the long run the best solution is to create a background server with delphi running under wine which the main linux server calls to do certain things it cannot do itself. Only those libraries go to the background server which cannot be replaced with fpc/lazarus equivalent. Everything else runs native under linux. Sometimes the background server may need to call some interfaces on the main server to do its job.
The problematic area with running a command line server under wine for me is making it to start after reboot. I could not find a way to autostart more then one server. But my linux knowledge is very limited. I thought maybe it should be started by the main server instead configuring Linux to do it, but could not get it working either. Did not try very long though.
ab,
this is excellent stuff! I wanted to do the same thing for security reasons. Even when accessing the server would not be a problem it is much more secure if the servers with sensitive data are not accessible directly from the outside world at all.
I am eagerly waiting for the addition of http(s) tunneling. ![]()
AB,
looking at your code TSQLVirtualTable is designed for more general usage, it does not have to be connected to a TSQLRecord, but the implementation is missing for other use cases. Eg TSQLVirtualTable.Create has parameters to define field names, but it is never used. For full functionality Create should it not have some additional parameters defining the datatypes, sizes and other parts of the create SQL statement (like "not null" ...) to be formed properly?
This may help to get started:
Sapiem,
Lazus + LAMW + moRMot can be an option. LAMW makes it possible to use the same native Android controls to build the UI you would use in Android Studio. Lack of debugging is a big difference though.
When VER3_1 is defined the getter for the Name property creates a NEW string and the pointer to this string is used from there, not the original pointer. The question is what happens to this new string, will it ever be freed?
"memory leak" / "free NamePtr^"?
That is the gist of it. Either way the the fix is inconsistent, returns different pointers depending on the compiler directive.
Sapiem,
If you mean accessing the database locally on the device then the answer is that a mORMot server cannot be compiled under Delphi for Android.
You can try to use Lazarus to create an .so with the mORMot engine to be deployed with the Delphi project. mORMot cross-platform client can be used to connect to it under Delphi. Lazarus lacks debugging for Android so the engine part is difficult to debug if it has an android specific issue. Otherwise you can connect to a server running on a PC where the server side can be debugged.
AB,
I think this is worth attention.
AB,
I am curious: since NamePtr is public anyway it would seem obvious to use it directly. What is the point for going through the string property and it's getter to get to a pointer at the end which is already available? FPC ShortString is not reference counted so the getter returns a new string with a different pointer. Where will the new string be freed? It seems like a potential memory leak. Or if this is intentional and the new string is freed somewhere properly then when VER3_1 is not defined it will free VMP^.NamePtr^ which is likely to cause problems.
Thanks! ![]()