mormot_base.lpk + zeos

George · 2021-02-28 21:34:11

I installed ZEOS package as project dependency (without lazarus tool pallet icons, just runtime package).
This does not require any additional lines in the compiler path (from the project parameters).
Tested and it works.

Next, i found, that mORMot does provide lpk either.
I've read Packages\README.md.
Removed package define "-dNOSYNDBZEOS", added zcomponent as package dependency.
During compilation, i get error:
SynDBZeos.pas(122,2) Fatal: Cannot open include file "Zeos.inc"

Do i still need to fill "Other unit files", "Include files" and "Libraries" both for mORMot and ZEOS?
In my mind, since i have already added zeos as mORMot package dependency, it should just compile, am i wrong?

ps: latest mORMot 1.8 and zeos 8.0 (patch branch)

Last edited by George (2021-02-28 21:43:53)

George · 2021-02-28 21:56:31

From mormot_base.lpk options, I've added relative path to folder where Zeos.inc located.
Now, i successfully compiled mormot_base.lpk, SynCommons works, as well as ZEOS.
While my project does not contain any additional strings in compiler options ("Other unit files", "Include files" and "Libraries").
All dependencies are installed just for project. Awesome!

It makes sense to add to Packages\README.md note about link to Zeos.inc folder for other users.

Last edited by George (2021-02-28 22:03:55)

mdbs99 · 2021-03-04 10:54:36

Hi,
You shouldn't add relative path to folder where Zeos.inc is located but, instead, add`zcomponent` package from ZeosLib into it (item 5 from README).

George · 2021-03-04 19:49:54

Hi.

instead, add`zcomponent` package from ZeosLib into it (item 5 from README).

I use zeos 8, there were some issues that recently was fixed.
With latest zeos and mormot 1.18 I don't have any package compile errors with disabled {$DEFINE USE_SYNCOMMONS} from zeos.inc.

If {$DEFINE USE_SYNCOMMONS} is enabled in zeos.inc, clean mORMot package compilation ("recompile clean") will fail.
SynDBZeos.pas(343,26) Error: Identifier not found "TZJSONComposeOptions"
---
At some point it compiled. What I've done:
1. mormot_base. Compile without any changes, to get compiled SynCommons.
2. zcomponent. compile without {$DEFINE USE_SYNCOMMONS}, then with it.
3. mormot_base. add zcomponent dep and compile.
4. mormot_base. remove NOSYNDBZEOS from package options and compile twice!

Issue comes because of circular dependencies.
For example, zcomponent require SynCommons from mormot_base that require zcomponent.
With clean package installation, how it should be resolved?) Maybe I don't get something about lazarus packages.

Last edited by George (2021-03-04 21:35:56)

EgonHugeist · 2021-03-05 05:48:10

Hi,
i never did test the mORMot packages. That's new for me. OT: What are the advantages having them?
Just a side node: Zeos8 from trunk has a designtime package: zcomponentdesign.lpk. IMHO you guys should use this for the LCL-IDE instead. The zcomponent.lpk package is just a runtime package inbetween. Maybe that helps?

George · 2021-03-05 12:18:56

What is a Lazarus package

A Lazarus package is a collection of units and components, containing information how they can be compiled and how they can be used by projects or other packages or the IDE.
Packages are ideal for sharing code between projects.

Like NPM in javascript world.
If your lazarus project specify dependencies, it gives clear understanding of which third party code you need and which version.
When package greatly organized it's easier to use it inside your project by just adding it as dependency.
In case when you have multiple projects and want to share some code - package may be great solution.

Lazarus package may be design time or runtime.
Runtime packages must be included as dependency from project tree, while design time packages require in-IDE installation.
Both package types are useful and worth using.

Also, lazarus have online package manager (with auto-update feature) and corresponding repository, where mORMot is also listed.
(I don't use this, and dont like idea that package can be updated automatically).

Last edited by George (2021-03-05 12:32:40)

mpv · 2021-03-05 13:25:43

George wrote:

Like NPM in javascript world.

Unfortunately - no. Some advantages of npm are:
1) a local installation of packages for project. So I can have a several project each one with the different version of some package.
2) an easy to use: install package npm install package_name, check for updates: npm outdated and so on
3) scoping (using @) and having a different sources for packages depends on scope - this allow to easy create an on-premissi npm registry
4) and so on so on

All of this is not available with lazarus packages, so it is not applicable for production usage where version management is important.

This is why I use a git sub-repositories to manage a dependencies (mORMot and Zeos reposiroties are added as a git sub-repos for each of my project) - this solves many problems for me.

George · 2021-03-05 15:38:17

Lazarus package system is not as good as NPM, for sure.

I found strange behaviour, that looks like a lazarus bug.
We can put our dependencies in project subfolder:

dependencies
dependencies\readme.md
dependencies\mormot

In readme file we can store all deps details, like path to github repo, revision (commit ID), etc.

Now if I open mormot_base.lpk from this subfolder and click "use" - "add to project", then package appears in project inspector as dependency.
lpi file indeed does not contain any information about place where package lives:

      <Item1>
        <PackageName Value="mormot_base"/>
      </Item1>

But, from project inspector, we can right click on mormot_base, select option "store file name as default for this dependency".
Save changes. now, lpi file contain proper relative path:

      <Item1>
        <PackageName Value="mormot_base"/>
        <DefaultFilename Value="..\dependencies\mormot\Packages\mormot_base.lpk"/>
      </Item1>

But for some reason, this path will not be used as default.
So if we have another package and repeat all steps, mormot_base package will be used from last opened path.
This is really looks like a bug or wrong behavior.

I've checked second way.
Created global dependencies folder, for multiple versions of each one:
lazarus-packages
lazarus-packages\mormot\master\ // <- only this one connected to git repo
lazarus-packages\mormot\1.18.6247\
lazarus-packages\mormot\1.18.6248\

Copied src files to proper folders and opened lpk from each one, configured lpk version to match SynopseCommit.inc.
Each time when we open lpk package, lazarus will remember package name + version + path (see menu - package - package links).

Removed dependencies from project and re-added it from from project inspector - right click on dependencies, find mormot_base and specify "minimum" and "maximum" version.
Now, each project request exact mormot_base version and lazarus know where it is (checked menu - packages - package links).
But for some stupid reason, Lazarus can't properly load requested package for second project.

I don't know how is it possible to make such strange logic.
Is there any chance that they can fix it?
https://bugs.freepascal.org/my_view_page.php

Last edited by George (2021-03-05 15:44:52)

mpv · 2021-03-05 16:14:38

We can put our dependencies in project subfolder
....
In readme file we can store all deps details, like path to github repo, revision (commit ID), etc.

git already have everything to manage a dependencies as sub-repositories, you do not need to create a redme file (which is hard to maintain).

cd yourProjectRepo
git init #if not already a git repository
mkdir libs
git submodule add git@github.com:synopse/mORMot.git libs/mormot
git add libs/mORMot
git add .gitmodules # here stored an information about from where we got a mormot
git commit -m "added a latest mormot trunk as submodule"

Let's switch to some revision of mormot

git checkout -b trysomeOldMormot
cd libs/mormot
git checkout a04b4c4a0
cd ../..
git add libs/mormot
git commit -m "switched to motmot on revision merged #379 "

now we have 2 milestone for our project:

git checkout master --recurse-submodules #out project with mormot trunk
git checkout trysomeOldMormot --recurse-submodules #out project with fixer revision of mormot

and we do not need a lpk at all

George · 2021-03-05 16:32:25

git already have everything to manage a dependencies as sub-repositories, you do not need to create a redme file (which is hard to maintain).

I use local gitlab instance, and just excluded dependencies\mormot from my repo.
Like NPM packages that not supposed to be inside repo, I used readme just to store some useful tips about used packages.

and we do not need a lpk at all

That's default option for mormot framework.
But I don't understand why packages work so strange, so i opened issue, will see what laz maintainers will say.

mpv, do you use zeos in same way without lpk?
I didn't found manual how to add zeos directly.

mpv · 2021-03-06 08:03:16

Yes, I use Zeos in the same way by adding it as a subrepo from unofficial github mirror

George · 2021-03-06 11:11:56

About multiple versions of package. I created ticket here 38587.
It turns out that this issue is a bug, and may be fixed in Lazarus 2.2 (or with minor update later).

And there is a workaround available.

Last edited by George (2021-03-06 11:16:04)

mdbs99 · 2021-03-07 12:03:43

George wrote:

"For example, zcomponent require SynCommons from mormot_base that require zcomponent."

Are you sure that Zeos is using SynCommons on its official repository? I don't update Zeos for a long time...
If yes, Zeos may use mORMot package instead to use SynCommons.pas direclty.

George · 2021-03-07 14:14:46

Are you sure that Zeos is using SynCommons on its official repository?

Yes, but it's optional and disabled by default.
Currently they even have mORMot 2 integration, that can be enabled form inc file.

Soon, Lazarus will get update with fix 38587. And then, I hope, Lazarus package control feature will work as expected.
If so AB would be able (if he wants) to rely on this features and create few packages for FPC users, at least, for mORMot2.
In order to solve circular dependencies, ZEOS can optionally rely on some common mORMot package, while mORMot common package will not require any zeos packages (even, when zeos integration in mORMot is switched on).
Additionally, every package should have proper version for each build (in order to work with Lazarus properly).

Last edited by George (2021-03-07 14:43:26)

ab · 2021-03-07 20:10:44

Thinking out loud:

What if we define a mormot2 package and a mormot2-zeos package (including only mormot.db.sql.zeos.pas) ?
Together with a zeos package and a zeos-mormot2 package.
The zeos-mormot2 package could have zeos and mormot2 as dependency, and mormot2-zeos both mormot2 and zeos as dependencies.

Or even just compile zeos with no MORMOT2 flag - and use this plain zeos package from mormot2-zeos.
I suspect the performance optimization may not be very high, since zeos is already well optimized.
If a project need the extra-performance of the MORMOT2 Zeos flag, it could use it explicitly within its build process - but for the development within the IDE, a plain zeos without MORMOT2 could be enough.

It could fix the dependency issue, I guess.

George · 2021-03-07 22:26:38

Currently, I'm not familiar with new mORMot2 module naming
But, I can share thoughts.

In order to compile zeos with mormot, zeos require some files, for example, SynCommons (which require SynLZ as well).
With those modules, zeos should compile.
So, you may create mormot2.common.lpk and include modules, that does not require much of your modules, just common part like SynCommon, SynLZ (and definitely, mormot.common.lpk should not rely on zeos).
In that case, zeos team can add custom package option (like you did) if checked, user should add mormot.common.lpk dependency to zcomponent package manually in order to compile zeos.

Next, your module, that rely on zeos, should be extracted to separate package, mormot2.zeos.lpk this one should require mormot2.common.lpk and appropriate zeos packages (zdbc, zcore).
Should be another main package, something like mormot.base.lpk (or mormot.core.lpk, sounds better) this one will require mormot2.common.lpk (and mormot.cross.lpk if this one exists in mORMot2 and is mandatory for mormot.base.lpk, otherwise mormot.cross.lpk may be standalone package).
When user compile mormot.base.lpk with enabled option "NOSYNDBZEOS" (it's better to change it to USESYNDBZEOS) then, mORMot should compile fine without any relations to zeos.
When user compile mormot.base.lpk with disabled option "NOSYNDBZEOS", he also must add mormot2.zeos.lpk dependency (mormot2.zeos.lpk will automatically require appropriate zeos packages, and Lazarus will try to find it).
Would be good to add mormot2.synode.lpk as well (that will require mormot2.common.lpk).

And last, but not least.
Every package should be properly versioned (with version data from SynopseCommit.inc).
Zeos team may also put package build number from commit code. SVN generates numerical codes. For example, latest is 7366, so, full package version may be 8.0.7366.

---
Later, when Lazarus 2.2 will be released, for zeos package dependencies, you can specify min and max versions that should work with mORMot2 (for example >= 8.0.xxxx).

---
By the way, conditional defines in zeos.inc allows to turn off some drivers and doing so, reduces resulting exe file (package must be recompiled).
Such switches are good and should present in package "custom options defines" as well. So, package will be more user friendly for tuning.

Last edited by George (2021-03-07 22:38:20)

mdbs99 · 2021-03-08 00:18:25

ab wrote:

Thinking out loud:
What if we define a mormot2 package and a mormot2-zeos package (including only mormot.db.sql.zeos.pas) ?
Together with a zeos package and a zeos-mormot2 package.
The zeos-mormot2 package could have zeos and mormot2 as dependency, and mormot2-zeos both mormot2 and zeos as dependencies.
Or even just compile zeos with no MORMOT2 flag - and use this plain zeos package from mormot2-zeos.
I suspect the performance optimization may not be very high, since zeos is already well optimized.
If a project need the extra-performance of the MORMOT2 Zeos flag, it could use it explicitly within its build process - but for the development within the IDE, a plain zeos without MORMOT2 could be enough.
It could fix the dependency issue, I guess.

We should create a "mormot.base.pack" (or mormot.base.run, following the "suffix pattern" which "run" stands for runtime packages and "dsgn" stands for IDE installed package).
Then, create another one just for Zeos, mormot.db.zeos.pack.
Zeos team should use mormot.base.pack instead .inc, conditionals, or paths for mORMot.

mdbs99 · 2021-03-08 00:19:49

IIRC, by the time, we didn't create a package for Zeos because units were not separated as they are now in version 2.

George · 2021-03-08 07:24:09

We should create a "mormot.base.pack" (or mormot.base.run, following the "suffix pattern" which "run" stands for runtime packages and "dsgn" stands for IDE installed package).
Then, create another one just for Zeos, mormot.db.zeos.pack.
Zeos team should use mormot.base.pack instead .inc, conditionals, or paths for mORMot.

So, Zeos will require mormot always?
mormot.base.pack is entire framework, not just common module. Will it affect to exe size?

Last edited by George (2021-03-08 07:24:21)

mdbs99 · 2021-03-09 00:48:37

So, Zeos will require mormot always?

I wouldn't say always, but if Zeos needs mORMot, this code should be provided from a package.

mORMot Open Source

#1 2021-02-28 21:34:11

mormot_base.lpk + zeos

#2 2021-02-28 21:56:31

Re: mormot_base.lpk + zeos

#3 2021-03-04 10:54:36

Re: mormot_base.lpk + zeos

#4 2021-03-04 19:49:54

Re: mormot_base.lpk + zeos

#5 2021-03-05 05:48:10

Re: mormot_base.lpk + zeos

#6 2021-03-05 12:18:56

Re: mormot_base.lpk + zeos

#7 2021-03-05 13:25:43

Re: mormot_base.lpk + zeos

#8 2021-03-05 15:38:17

Re: mormot_base.lpk + zeos

#9 2021-03-05 16:14:38

Re: mormot_base.lpk + zeos

#10 2021-03-05 16:32:25

Re: mormot_base.lpk + zeos

#11 2021-03-06 08:03:16

Re: mormot_base.lpk + zeos

#12 2021-03-06 11:11:56

Re: mormot_base.lpk + zeos

#13 2021-03-07 12:03:43

Re: mormot_base.lpk + zeos

#14 2021-03-07 14:14:46

Re: mormot_base.lpk + zeos

#15 2021-03-07 20:10:44

Re: mormot_base.lpk + zeos

#16 2021-03-07 22:26:38

Re: mormot_base.lpk + zeos

#17 2021-03-08 00:18:25

Re: mormot_base.lpk + zeos

#18 2021-03-08 00:19:49

Re: mormot_base.lpk + zeos

#19 2021-03-08 07:24:09

Re: mormot_base.lpk + zeos

#20 2021-03-09 00:48:37

Re: mormot_base.lpk + zeos

Board footer