Revision 2.x of the framework

ab · 2020-03-07 13:14:06

@danielkuettner
Version 2 initial goal is to be 100% compatible with version 1.18 - once the uses clauses have been changed.
For instance, instead of "uses SynCommons.pas", write "uses mormot.core.base, mormot.core.text, mormot.core.json".
Then all internal code and classes would remain the same.
Even if we do refactoring of the classes, TSQLRest compatibility interface would remain, just calling new nested classes by composition.
Goal is to have all 1.18 samples working with no modification - with the exception of the VCL UI part, which is not migrated at first (but I guess it is not widely used).

@mpv
This is good news.
So we could use gitlab.com instead of github.com and have CI in the Free plan?
Nice!

danielkuettner · 2020-03-07 13:35:12

@AB
Good news!

But we still have to look for the new place of used functions/classes etc. Are you able to give infos about old vs. new places?

mpv · 2020-03-07 14:29:01

So we could use gitlab.com instead of github.com and have CI in the Free plan?

Yes, even migration can be done automated

This is how gitlab merge request for big file looks like: https://gitlab.com/pavel.mash/testexter … rallel&w=1
Take attention to "Show unchanged lines" links (this feature is missing on github)
In case of sveral files we have navigation on left side

All changes (and conflict resolving) can be done inplace right from Web UI

In CI we can add automatic mirroring gitlab project to read-only github version (after MS buy a github many FOSS project did such)

Last edited by mpv (2020-03-07 14:32:26)

ab · 2020-03-07 14:34:30

@danielkuettner
We just started the work. Only 20% of SynCommons.pas is converted now... still a lot to do!
My goal is to run all regression tests and all samples with no modification, but the "uses" clause.

@mpv
GitLab seems like a good alternative. Especially the CI stuff, which could be enough for mORMot2.
How do we deal with static .o/.dll files? With GitHub we could use the files attached to the Releases, and not include those binaries in the main git source code repository - how can we do this with GitLab?

mpv · 2020-03-07 15:06:05

How do we deal with static .o/.dll files?

In my private gitlab we add it as a git LFS. With github this is a PITA (may be you remember I try to did such some times ago). I don`t know is there any hidden problems with LFS on GitLab.com, on the features page LFS is on the free plan.

I adds all mORMot binary as LFS into test project (uploaded for 8 second - lets wait what heppens - see https://gitlab.com/pavel.mash/testexter … ts/1/diffs

Last edited by mpv (2020-03-07 15:06:46)

urhen · 2020-03-08 22:44:53

You can also run CI on github. There are different possibilities but I'm not sure if any supports FPC. Unfortunately that's a big disadvantage in FPC/Delphi community because nobody really cares about such basic but important features.
To add FPC to Travis CI you need at least 3 volunteers (https://docs.travis-ci.com/user/languag … languages/). But you could use a Docker image for your CI.
Another option would be Github Actions (https://github.com/features/actions) or AppVeyor (https://forum.lazarus.freepascal.org/in … ic=35060.0). Example: https://forum.lazarus.freepascal.org/in … #msg190018
I think I've also seen people who trigger their own CI server via Github.

Last edited by urhen (2020-03-08 22:54:53)

urhen · 2020-03-08 22:51:16

Maybe adding an EditorConfig (https://editorconfig.org/) file would also be good for people who use the web IDE to submit small patches or Lazarus/Visual Studio Code.

edwinsn · 2020-03-09 04:19:55

With github I can contribute changes very easily and I believe many are familiar with the process. How the gitlab's process compared to github's Fork-Clone-Edit-Commit-CreatePR process?

snakegao · 2020-03-09 04:20:00

Does the version 1.1.8 still update or stop when Revision 2.x of the framework starts ?

ab · 2020-03-09 11:43:58

My idea was to have a smooth transition from 1.18 to 2.x.

In four steps:

A) 2.0 will be feature-identical to 1.18.
When all existing tests pass with 2.0, the only thing to change in your code base would be the unit names in "uses" clauses. We could make a quick search/replace for it.
Only spliting main units would be implemented at that time.

B) Then, we will probably maintain fixes to 1.18 for some time (a few months), but not adding any feature, nor make any optimization.
Once 1.18 would be known to be as stable as possible, it would freeze.

C) Every active project should eventually move to 2.x, and its supported compilers - which should be painless if A) is made correctly.

D) Version 2.x will then starting be refactored, e.g. TSQLRest monolitic classes would be de-composed as nested classes with proper SOLID separation: e.g. TSQLRest.orm or TSLQRest.soa - which would be implemented in dedicated units.
The TSQLRest current methods (too many to my taste) would be marked as deprecated and nested sub-classes should rather be directly called. They will be defined as "inlined" wrappers, with no more documentation (which would move to the nested classes), to show they are deprecated. In the user code, you would be encouraged to write rest.orm.Add() instead of rest.Add() for instance. A conditional define would be used to disable the deprecated wrappers, and force identifying the code to be changed at compile time.

Survey update: we are close to 100 responses, so in the next days I would disclose the result and we all will make some decisions, e.g. about Delphi versions officially supported (which are likely to be Delphi 7, 2007 and 10.3 - the other versions being supported by contributors) and how we handle the source (github or gitlab?).

macfly · 2020-03-09 11:57:33

A) 2.0 will be feature-identical to 1.18.
When all existing tests pass with 2.0, the only thing to change in your code base would be the unit names in "uses" clauses. We could make a quick search/replace for it.
Only spliting main units would be implemented at that time.
...
D) Version 2.x will then starting be refactored...

I thought that version 2.x would already be implemented with all changes and compatibility breaks. Which in my opinion would be correct.

Wouldn't this be the opposite of what the new version control system specifies?

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Last edited by macfly (2020-03-09 11:58:02)

EMartin · 2020-03-09 12:09:06

I agree with @macfly, version 2 should be breaking change this way the migration will require the source code update.

on the other hand, ORM data response could be a class that could be replaced by other, for example for return in OData format ?

Thanks.

ab · 2020-03-09 12:09:58

The API will be broken for sure. Existing code won't compile until you change your "uses" clauses.
This is an incompatible API change.
But we won't break for the perverse need of breaking: we would like to have a smooth transition.

macfly · 2020-03-09 12:20:49

I agree with the smooth transition.

I am just commenting that this goes against the new specification that will be adopted.

This way it looks like a preparatory version for 2.x. Something like 1.19.x

This, of course, if the Semantic Versioning is followed. But nothing prevents postponing the adoption of this format.

mpv · 2020-03-09 12:22:33

With github I can contribute changes very easily and I believe many are familiar with the process. How the gitlab's process compared to github's Fork-Clone-Edit-Commit-CreatePR process?

The workflow is the same except "Pull Request" in GitLab is named "Merge Request" (which is better IMHO) and merging UI have better UX

sakura · 2020-03-09 13:21:17

macfly wrote:

I am just commenting that this goes against the new specification that will be adopted.

Call it 2.0.0.x Alpha-Release and breaks are okay, until you come to the release, I would say ;-)

macfly · 2020-03-09 13:42:51

For me it can be called anything, for example. supper.mORMot.with.steroids.

I commented, because the specification makes it clear that version change occurs when api compatibility is broken.

Compatibility break => New version => Changes that do not break compatibility in this version

What will be done in this case is to first change the version and then implement the compatibility break.

New version => Changes that do not break compatibility => compatibility break that do not change version.

The meaning is reversed.

But I repeat, it can be called anything, any version that is quiet for me.

Forget about it.

edwinsn · 2020-03-09 15:12:11

macfly wrote:

I agree with the smooth transition.
This way it looks like a preparatory version for 2.x. Something like 1.19.x

I disagree. ab is trying to achieve the migration path as smooth as possible, and that's the key for people to follow.

It's not like 1.19.x, while ab already planed to make changes like splitting TSQLRest into smaller pieces such as 'orm', 'rest', and so on, into different units. It'll already be API breaking changes, but just very logical, reasonable and easy to follow. And I appreciate ab's plan.

macfly · 2020-03-09 16:03:16

@edwinsn,

I agree with the smooth transition also. You have quotede this

But, first changing the version and then applying the API break is the opposite of what the proposed new versioning (https://semver.org) specifies.

That's why I proposed to apply the new versioning later, If it is really important to adopt him.

Last edited by macfly (2020-03-09 16:06:13)

George · 2020-03-09 16:36:12

TortoiseGIT - nice git client, works perfectly with gitlab as well.
I use it with beyond compare. Worth to try.

edwinsn · 2020-03-10 03:59:28

I've not used gitlab before, but the majority of the World is using github, it must for a good reason.

esmondb · 2020-03-10 07:35:51

I like fossil. It’s simple.

mpv · 2020-03-10 09:29:04

The majority of the World is using github, it must for a good reason

The reason is very simple - GitHub were the first. After M$ but GitHub many Open Source projects migrate to GitLab. Also with GitLab we can build a CI for mORMot.

Fossil is really simple but IMHO for open source project like mORMot collaboration is very important feature. With GitLab/GitHub contribution became much easy (pull requests, code review, comments etc.). More developers can made a PR (as far as I remember just a few persons did it with fossil)

And there is another feature (may be this is a most important feature for OSS projects) - both GitLab/GitHub shows who did a Pull Request, so developer can show activity on Git**b for colleague/employer/customer or for himself and say "I'm contribute to project xyz. My code is used by other peoples. I do something what made a world better ". I think this is a motivation for many of us.

And tooling - git integrated with many IDE, even with Explorer (TotroiseGIT), and have 100500 other tool what made developer life easy. Person who do not know git can easy found a tutorial for beginner or for experts and so on.

urhen · 2020-03-10 22:31:21

From point of use github/gitlab are more or less identical, just a different look and naming. Some things are better at Github, some at Gitlab.
But I think more people still use Github than Gitlab (more users have account on Github than Gitlab).
If the only thing is the runner, you could also mirror from Github to Gitlab and have the runner there (probably only important to the few devs anyway). I'm sure you can also use your private runner with Github if you configure everything as you'd do for Gitlab.

macfly · 2020-03-11 00:17:18

Well, if one of the goals of the new version is to facilitate adoption, I think github is the best option.

edwinsn · 2020-03-11 03:05:42

I agree, I suggest continue using github if possible. "Everyone has an account and is familiar with" makes a lot of differences!

TPrami · 2020-03-12 09:42:58

mpv wrote:

Nice!
BTW size of SynCommons is not a problem for Lazarus, only for Delphi.

Size of problem for Programmers mainy, easy to start new project, couple of units in uses and that's it. But pain to go through, be SyCommons have lot of stuff most of users have not even seen

I would prefer unit scopes/namespaces but that would mean dropping quite many delphi versions (dotted filenames) mORMot.JSON.Utils /mORMot.UF8.Utils ... (or similar)

For older delphi versions there would always be 1.18, so not the worst thing to drop all but pretty much patest delphi versions.

-Tee-

ab · 2020-03-12 10:08:42

Dotted names do work on Delphi 7.
It has no problem with units named e.g. mormot.core.base.pas.
Check and try https://github.com/synopse/mORMot2

What do not work is unit aliases/namespaces - but we don't need them in our case.

TPrami · 2020-03-12 11:28:11

ab wrote:

I am preparing the new repository, and start naming things and folders.
Check https://github.com/synopse/mORMot2
We propose to switch to dotted named units, e.g. mormot.core.json.pas

WOOOHOO!!!!

macfly · 2020-03-12 18:21:45

@ab, Will you document changes to the unit names? Or provide the "from" "to" somewhere?.

Maybe keep an INI file inside the repository with the all "search => replaces".

Example

syncommons=mormot.core.base, mormot.core.json
synxyz=mormot.xyz

I remember that the JVCL has a tool that searches and replaces in all defined files/subdirs.
And it supports loading an INI file with definitions.

ab · 2020-03-12 19:36:50

@macfly
Yes I had something like that in mind.
In initial 2.0 revision, there will be easy search/replace function as you propose.

macfly · 2020-03-12 19:49:53

@ab, very good. Thanks for all

urhen · 2020-03-14 22:59:10

I think the issue on mORMot2 project about creating only a new branch is a good advice!
I mean, you can still start from 0 on the new branch and if your finished you fork the current master branch to mormot-old and delete the master branch. After that your new mormot2 branch gets renamed to master branch.
All stars, follower etc will remain but you achieved the same result as with creating a new project

ab · 2020-03-15 17:55:20

Yes, I guess I will make a branch, or at least put back mORMot2 into the main mORMot repository.
It would be less confusing.
But I won't merge the two until mORMot2 is finalized and able to replace current revision.

itSDS · 2020-03-16 09:39:27

I like your upgrade thoughts and filled out the survey
What i was missing is : will complete CrossPlatform support available with 10.4 ? (They change something with arc ?)

ab · 2020-03-16 09:47:40

I guess SynCrossPlatform* code is both ARC and non-ARC compatible, so should work as it is.

urhen · 2020-03-29 13:30:27

Do you plan to release the results from the Survey?

ab · 2020-03-29 17:42:35

Yes, I will make an article tomorrow.

We got 155 responses, which is a nice number.
I guess the direction will be clearer.

Thanks you all!

pvn0 · 2020-03-30 09:00:34

I've got a suggestion for revision 2.0,

I was just doing some tests using SynCommons.ObjectToJSON and despite what I remember and what documentation said, it couldn't serialize a simple object. Turns out that if you don't have mORMot.pas in your uses clause the DefaultTextWriterSerializer is assigned a gimped version which can only serialize certain lists and collections, unlike the more advanced serializer from mORMot.pas which works directly with RTTI. Hopefully with refactoring in revision 2.x, this kind of anti patterns will disappear.

ab · 2020-03-30 12:34:48

In mORMot2, class serialization would not be in the ORM unit, but in a dedicated mormot.core.json unit.
This will be part of the refactoring.

pvn0 · 2020-03-30 13:13:47

That's good news, thank you ab!

urhen · 2020-04-16 22:48:14

Hey ab,
maybe you should consider splitting the stuff into more files because e.g. mormot.core.base.pas almost has 10k lines again. I know that there is also a lot of documentation but I mean it seems your still somewhere at the beginning of the refactoring and most stuff is still to come...

macfly · 2020-04-16 23:47:04

The new formatting style (which is better) also generates more lines.

ab · 2020-04-17 17:43:53

MacFly is right: new style increases the number of lines for sure.

mormot.core.base.pas won't grow much any more. Note that 10K is excluding the asm included files (around 2K to add).
mormot.core.data.pas has 13K lines.
mormot.core.text.pas has 15K lines.

Note that on Delphi, having more units tends to slow down everything - at least without Andy IDE FixPack. And too much units would make the framework impossible to work with (huge "uses" clause in units, for instance).
Also note that we can't make small files in object pascal as we can e.g. in C#, since we don't have partial classes. And I always hated the one class = one file paradigm of Java/C#.
So we try to make a balance between number of units and size of each unit.

I guess 10K-15K lines as maximum size is fair enough - it is smaller than some RTL units (sysutils.pas was 17K in Delphi 7, it is 35K in 10.3).
The main problem is about units interdependence. The new mORMot2 unit design has a much cleaner approach, thanks to better separation of concerns, so once one unit reaches its compilation size, it would not increase in the future. See for instance how much is in mormot.core.rtti.pas (around 6K lines) - so it is not only a TypInfo.pas wrapper, but a root unit for advanced RTTI cache and customization with the TRttiCustom abstract class.
We reduced also the code size thanks to its better design. For instance, we have a single JSON serialization kernel, and a single class RTTI cache in mormot.core.json.pas, whereas we had at least 4 in mORMot 1.18: one in SynCommons.pas (record/arrays) and three in mORMot.pas: ORM + objects + SOA... Not only there is less code, and more consistent feature-set, but also better performance: we optimize once for all.

macfly · 2020-05-02 14:14:20

If anyone else missed it, how did I ...
I saw today in the FastMM5 topic that the Poll result is on the blog.

http://blog.synopse.info/post/2020/03/3 … ot2-Survey

Since 3/30

edwinsn · 2020-05-13 11:26:33

Hey! I've got one thing to add:

Do you plan to eliminate the use of the 'with' statement of the Object Pascal language?

Thanks.

ab · 2020-05-13 12:14:51

"with" is not always evil, when the scope is very small and well tested.

emk · 2020-05-13 13:07:51

What @ab said + "with" helps compiler to make some optimization.

PBa · 2021-01-10 11:44:19

@ab:
in mORMot2 survey you also asked how to continue with source code management and there were many answers to also keep fossil as scm (at least in parallel to github). Just for curiosity: do you intend to set up also a fossil repository for mORMot2 in future?
Thanks a lot and have a nice weekend!
Paul

okoba · 2021-01-10 12:20:18

@ab I noticed you broke the propertied to two lines, can I ask why? Is it a readably improvement or something else?

mORMot Open Source

#51 2020-03-07 13:14:06

Re: Revision 2.x of the framework

#52 2020-03-07 13:35:12

Re: Revision 2.x of the framework

#53 2020-03-07 14:29:01

Re: Revision 2.x of the framework

#54 2020-03-07 14:34:30

Re: Revision 2.x of the framework

#55 2020-03-07 15:06:05

Re: Revision 2.x of the framework

#56 2020-03-08 22:44:53

Re: Revision 2.x of the framework

#57 2020-03-08 22:51:16

Re: Revision 2.x of the framework

#58 2020-03-09 04:19:55

Re: Revision 2.x of the framework

#59 2020-03-09 04:20:00

Re: Revision 2.x of the framework

#60 2020-03-09 11:43:58

Re: Revision 2.x of the framework

#61 2020-03-09 11:57:33

Re: Revision 2.x of the framework

#62 2020-03-09 12:09:06

Re: Revision 2.x of the framework

#63 2020-03-09 12:09:58

Re: Revision 2.x of the framework

#64 2020-03-09 12:20:49

Re: Revision 2.x of the framework

#65 2020-03-09 12:22:33

Re: Revision 2.x of the framework

#66 2020-03-09 13:21:17

Re: Revision 2.x of the framework

#67 2020-03-09 13:42:51

Re: Revision 2.x of the framework

#68 2020-03-09 15:12:11

Re: Revision 2.x of the framework

#69 2020-03-09 16:03:16

Re: Revision 2.x of the framework

#70 2020-03-09 16:36:12

Re: Revision 2.x of the framework

#71 2020-03-10 03:59:28

Re: Revision 2.x of the framework

#72 2020-03-10 07:35:51

Re: Revision 2.x of the framework

#73 2020-03-10 09:29:04

Re: Revision 2.x of the framework

#74 2020-03-10 22:31:21

Re: Revision 2.x of the framework

#75 2020-03-11 00:17:18

Re: Revision 2.x of the framework

#76 2020-03-11 03:05:42

Re: Revision 2.x of the framework

#77 2020-03-12 09:42:58

Re: Revision 2.x of the framework

#78 2020-03-12 10:08:42

Re: Revision 2.x of the framework

#79 2020-03-12 11:28:11

Re: Revision 2.x of the framework

#80 2020-03-12 18:21:45

Re: Revision 2.x of the framework

#81 2020-03-12 19:36:50

Re: Revision 2.x of the framework

#82 2020-03-12 19:49:53

Re: Revision 2.x of the framework

#83 2020-03-14 22:59:10

Re: Revision 2.x of the framework

#84 2020-03-15 17:55:20

Re: Revision 2.x of the framework

#85 2020-03-16 09:39:27

Re: Revision 2.x of the framework

#86 2020-03-16 09:47:40

Re: Revision 2.x of the framework

#87 2020-03-29 13:30:27

Re: Revision 2.x of the framework

#88 2020-03-29 17:42:35

Re: Revision 2.x of the framework

#89 2020-03-30 09:00:34

Re: Revision 2.x of the framework

#90 2020-03-30 12:34:48