#1 2021-12-20 08:28:14

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

mORMot 2 Generics and Collections

Generics are a clever way of writing some code once, then reuse it for several types.
They are like templates, or compiler-time shortcuts for type definitions.

In the last weeks, we added a new mormot.core.collections.pas unit, which features:

* JSON-aware IList<> List Storage;
* JSON-aware IKeyValue<> Dictionary Storage

In respect to Delphi or FPC RTL generics.collections, this unit uses interfaces as variable holders, and leverage them to reduce the generated code as much as possible, as the Spring4D 2.0 framework does, but for both Delphi and FPC. It publishes TDynArray and TSynDictionary high-level features like indexing, sorting, JSON/binary serialization or thread safety as Generics strong typing.

Resulting performance is great, especially for its enumerators, and your resulting executable size won't blow up as with the regular RTL unit.

This is the conversation thread for the https://blog.synopse.info/?post/2021/12 … ollections blog post.

Offline

#2 2021-12-20 10:26:21

zed
Member
From: Belarus
Registered: 2015-02-26
Posts: 105

Re: mORMot 2 Generics and Collections

What is the minimum supported Delphi version for this? Is it 2009?

Offline

#3 2021-12-20 12:10:48

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

Since Delphi XE, but we had internal errors on XE6 and XE7 so they are undefined for those.
I have added this information at the beginning of the unit.

Ahead of time specialization is only available since XE8 because it generates internal errors with earlier compilers...

So, a recent Delphi version (10.x) is recommended.

In a nutshell, Delphi generics support is very inconsistent and workarounds are difficult to find out.
We already spent hours fighting against Delphi compiler bugs. If someone if willing to find workarounds for some unsupported compilers, they are welcome to make pull requests.

Offline

#4 2021-12-20 13:34:32

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Does it need to register a new class like the following in order to work?

  Ttest=class
  private
    fsoup:string;
  published
    property soup:string read fsoup write fsoup;
  end;

v:=collections.NewList<Ttest>([loCreateUniqueIndex],typeinfo(tarray<Ttest>));

Offline

#5 2021-12-20 14:18:14

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

No need to register anything.

v:=collections.NewList<Ttest>([loCreateUniqueIndex]);

should be enough - no need of TypeInfo().

Offline

#6 2021-12-20 14:34:24

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Running the above statement you showed for the Ttest class, and having v:ilist<Ttest>, I am getting:
Project Project1.exe raised exception class ESynList with message 'TSynListSpecialized<System.Integer>.Create: T does not match TArray<System.Integer>'.

In line 777 of mormot.collections,

     (fDynArray.Info.ArrayRtti.Kind <> aItemTypeInfo^.Kind))  then

The first rkInteger and the second is rkClass

Last edited by dcoun (2021-12-20 14:34:50)

Offline

#7 2021-12-20 14:47:59

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

You are right.
I have just introduced a small regression.

I will fix this.

Offline

#8 2021-12-20 15:09:02

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Thank you very much Arnaud.
playing a bit with it, I can say that an "Exchange" between two items could be useful too (if loCreateUniqueIndex is not used).

Last edited by dcoun (2021-12-20 15:12:07)

Offline

#9 2021-12-20 15:46:08

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

Please check with https://github.com/synopse/mORMot2/commit/3600e919

Now, setting the dynamic array type is not needed: it will properly use TArray<T>.
So I guess the parameter order is good too.

Offline

#10 2021-12-20 16:14:37

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

In the following the returned i is -1
if I remove the loCreateUniqueIndex, I am getting i=0, but the following

v.items[i]

is null
Delphi 11 and FPC 3.2

var i:integer; s:Ttest; 
begin
v:=collections.NewList<Ttest>([loCreateUniqueIndex]);
s:=Ttest.Create; 
s.soup:='hello';
i:=v.Add(s);
showmessage(v.items[i].soup);

Offline

#11 2021-12-20 21:00:23

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

loCreateUniqueIndex is unsupported for collections.

We may compare the pointers, but it is of little use in practice, isn't it?
I will enable pointer comparison.

Offline

#12 2021-12-21 07:05:02

edwinsn
Member
Registered: 2010-07-02
Posts: 1,215

Re: mORMot 2 Generics and Collections

Great! Spring4D is great too but in most cases I don't want to introduce another not-so-small dependency, and now we have generic collections in mORMot2 too!

I like that the methods are directly attached to IList and IKeyValue. On the other hand, with Spring4D you have a LOT of interfaces and types to explorer in order to find what you need.

One question - What's the **best practice** to implement a reusable descendant of IList<T>?

With my past experiences of using **interface-based** collections, I won't say I like it very much, because, for example, with TObjectList, I can easily create an extended TObjectListEx, which I add a lot of new methods.
Let's face it - we don't have very much situations where we need a list to treat the Interger, String, Doubles, Objects, and so on, with the exactly same logic. Sometimes, it's just **too generic**.
So after years of using spring4D, I go back to use my TObjectList descendant, because in the end I figured out that all I need is just "generic collections with powerful features (loop, some, filter, etc) packed without bloating the EXE".

Hope it makes sense.


Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.

Offline

#13 2021-12-21 07:32:13

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

The idea is to encapsulate the list within another regular class, if you need to add functionalities.
You use the list or the key/value for storage, then you add your logic on top of it.
This is what we did a lot with TSynDictionary or TDynArray in the framework.
My advice is to use composition rather than inheritance: use a fList: IList<> instead of inheriting from the list.

Otherwise, an IList<somerecord> or an IList<TsomeORM> could be useful to store some working process information, just as it is.
Behind the scene, it is just one (or two) dynamic arrays, which you access througout the methods.

Offline

#14 2021-12-21 08:36:15

edwinsn
Member
Registered: 2010-07-02
Posts: 1,215

Re: mORMot 2 Generics and Collections

Yes, "composition over inheritance" wink


Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.

Offline

#15 2021-12-21 10:09:59

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

Offline

#16 2021-12-21 10:25:41

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Now the object is added but when trying to access it with the last line in the code above the result for 0-index items is nil

Offline

#17 2021-12-21 13:43:09

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

Try to debug why it is the case on your side because I can't reproduce it here.

Do the regression tests pass on your side?
There is a test for this exact case in https://github.com/synopse/mORMot2/commit/2758a7aa

Offline

#18 2021-12-21 21:30:33

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

ab wrote:

Try to debug why it is the case on your side because I can't reproduce it here.

Do the regression tests pass on your side?
There is a test for this exact case in https://github.com/synopse/mORMot2/commit/2758a7aa

it works OK using the following and TobjectwithID instances:

v:=collections.NewList<TobjectwithID>([loCreateUniqueIndex]); 

using the following type instances it does not

Ttest=class
  private
    fsoup:string;
  published
    property soup:string read fsoup write fsoup;
  end;

Should I register them before using them?

Last edited by dcoun (2021-12-21 21:31:15)

Offline

#19 2021-12-21 21:31:41

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

Hi,  (D10.4.2 ent)

Is it possible to use IList<> for records ?
It reports Type is too complex.

thank you,
Vojko

Offline

#20 2021-12-22 07:22:05

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Adding the following to test.core.collections with the above Ttest type definition and it fails

  
  lt: ilist<ttest>;
  t:ttest;
.....
  lt := Collections.NewList<Ttest>;
  lt.Capacity := MAX + 1;
  for i := 0 to MAX do begin
    t:=ttest.Create; t.soup:=inttostr(i);
    Check(lt.Add(t) = i);
  end;
  for i := 0 to lo.Count - 1 do
    Check(lt[i].soup = inttostr(i));

I have noticed also that when adding in the following screen, value is none in add function
https://pasteboard.co/U3hiCtiA6j7N.png

Last edited by dcoun (2021-12-22 07:28:39)

Offline

#21 2021-12-22 11:02:10

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

@dcount
You are right, TObjectWithID worked by chance.
I think I have fixed the issue about classes with https://github.com/synopse/mORMot2/commit/09e135f4

@VojkoCendak
The error message is explicit: you should use Collections.NewPlainList<> factory instead.
Also ensure you have the latest source code revision.

Offline

#22 2021-12-22 12:33:57

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 392

Re: mORMot 2 Generics and Collections

Thank you very much Arnaud. It works now OK

Offline

#23 2021-12-22 14:09:17

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

@ab

Yes it works with plain records. (managed not yet wink );


thank you

Offline

#24 2023-04-18 07:15:06

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

Hi,

Is it possible to convert to loCreateUniqueIndex after mylist := Collections.NewList<TDateTime> creation ?
maybe: mylist.UniqeIndex := True;


thank you

Last edited by VojkoCendak (2023-04-18 07:15:37)

Offline

#25 2023-04-18 07:18:11

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

hi,

is it possible to:
1. copy/insert/add from another list in one line ?

thank you

Offline

#26 2023-04-18 07:42:04

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

Hello,

1) no, the loCreateUniqueIndex option could be set only in the factory mehod, at creation.

2) yes, you can, accessing the TDynArray from the IList<T>.Data methods, so you can call TDynArray.AddDynArray().
I have added a new AddFrom() method to ensure we keep strong typing of the source list.
https://github.com/synopse/mORMot2/commit/8e541fb8

Offline

#27 2023-04-18 08:14:48

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

hi,

What about hash performance when it is copied from another list?

listnormal := Collections.NewList<TDateTime>;
listnormal gets populated ...
// want to copy from normal list to fast
listfast := Collections.NewList<TDateTime>([loCreateUniqueIndex]);
listfast.AddFrom(casi); // copy 
   // lots of iterations in some loop
    if listfast.IndexOf(cas)<0 then  // is slower when using AddFromversus just looping through listnormal and .Add
      anotherlist.add(cas);

Do we need to rehash it?

Last edited by VojkoCendak (2023-04-18 08:19:28)

Offline

#28 2023-04-18 08:26:16

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

yes,

I've added

  if Assigned(Another) then begin
    fDynArray.AddDynArray(Another.Data, Offset, Limit);
    if (fHasher <> nil) then
    	fHasher^.ForceReHash;
  end;

and it works

Offline

#29 2023-04-18 13:56:06

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,182
Website

Re: mORMot 2 Generics and Collections

My guess is that Hasher should NOT be handled as such because you may have some unexpected duplicates in ForceRehash.

Please try https://github.com/synopse/mORMot2/commit/51627ba2
(not fully tested yet)

Offline

#30 2023-04-18 14:15:04

VojkoCendak
Member
From: Celje Slovenia
Registered: 2012-09-02
Posts: 88

Re: mORMot 2 Generics and Collections

I see, very nicely spotted.

thank you

Offline

Board footer

Powered by FluxBB