mORMot 2 Generics and Collections

ab · 2021-12-20 08:28:14

Generics are a clever way of writing some code once, then reuse it for several types.
They are like templates, or compiler-time shortcuts for type definitions.

In the last weeks, we added a new mormot.core.collections.pas unit, which features:

* JSON-aware IList<> List Storage;
* JSON-aware IKeyValue<> Dictionary Storage

In respect to Delphi or FPC RTL generics.collections, this unit uses interfaces as variable holders, and leverage them to reduce the generated code as much as possible, as the Spring4D 2.0 framework does, but for both Delphi and FPC. It publishes TDynArray and TSynDictionary high-level features like indexing, sorting, JSON/binary serialization or thread safety as Generics strong typing.

Resulting performance is great, especially for its enumerators, and your resulting executable size won't blow up as with the regular RTL unit.

This is the conversation thread for the https://blog.synopse.info/?post/2021/12 … ollections blog post.

zed · 2021-12-20 10:26:21

What is the minimum supported Delphi version for this? Is it 2009?

ab · 2021-12-20 12:10:48

Since Delphi XE, but we had internal errors on XE6 and XE7 so they are undefined for those.
I have added this information at the beginning of the unit.

Ahead of time specialization is only available since XE8 because it generates internal errors with earlier compilers...

So, a recent Delphi version (10.x) is recommended.

In a nutshell, Delphi generics support is very inconsistent and workarounds are difficult to find out.
We already spent hours fighting against Delphi compiler bugs. If someone if willing to find workarounds for some unsupported compilers, they are welcome to make pull requests.

dcoun · 2021-12-20 13:34:32

Does it need to register a new class like the following in order to work?

  Ttest=class
  private
    fsoup:string;
  published
    property soup:string read fsoup write fsoup;
  end;

v:=collections.NewList<Ttest>([loCreateUniqueIndex],typeinfo(tarray<Ttest>));

ab · 2021-12-20 14:18:14

No need to register anything.

v:=collections.NewList<Ttest>([loCreateUniqueIndex]);

should be enough - no need of TypeInfo().

dcoun · 2021-12-20 14:34:24

Running the above statement you showed for the Ttest class, and having v:ilist<Ttest>, I am getting:
Project Project1.exe raised exception class ESynList with message 'TSynListSpecialized<System.Integer>.Create: T does not match TArray<System.Integer>'.

In line 777 of mormot.collections,

     (fDynArray.Info.ArrayRtti.Kind <> aItemTypeInfo^.Kind))  then

The first rkInteger and the second is rkClass

Last edited by dcoun (2021-12-20 14:34:50)

ab · 2021-12-20 14:47:59

You are right.
I have just introduced a small regression.

I will fix this.

dcoun · 2021-12-20 15:09:02

Thank you very much Arnaud.
playing a bit with it, I can say that an "Exchange" between two items could be useful too (if loCreateUniqueIndex is not used).

Last edited by dcoun (2021-12-20 15:12:07)

ab · 2021-12-20 15:46:08

Please check with https://github.com/synopse/mORMot2/commit/3600e919

Now, setting the dynamic array type is not needed: it will properly use TArray<T>.
So I guess the parameter order is good too.

dcoun · 2021-12-20 16:14:37

In the following the returned i is -1
if I remove the loCreateUniqueIndex, I am getting i=0, but the following

v.items[i]

is null
Delphi 11 and FPC 3.2

var i:integer; s:Ttest; 
begin
v:=collections.NewList<Ttest>([loCreateUniqueIndex]);
s:=Ttest.Create; 
s.soup:='hello';
i:=v.Add(s);
showmessage(v.items[i].soup);

ab · 2021-12-20 21:00:23

loCreateUniqueIndex is unsupported for collections.

We may compare the pointers, but it is of little use in practice, isn't it?
I will enable pointer comparison.

edwinsn · 2021-12-21 07:05:02

Great! Spring4D is great too but in most cases I don't want to introduce another not-so-small dependency, and now we have generic collections in mORMot2 too!

I like that the methods are directly attached to IList and IKeyValue. On the other hand, with Spring4D you have a LOT of interfaces and types to explorer in order to find what you need.

One question - What's the **best practice** to implement a reusable descendant of IList<T>?

With my past experiences of using **interface-based** collections, I won't say I like it very much, because, for example, with TObjectList, I can easily create an extended TObjectListEx, which I add a lot of new methods.
Let's face it - we don't have very much situations where we need a list to treat the Interger, String, Doubles, Objects, and so on, with the exactly same logic. Sometimes, it's just **too generic**.
So after years of using spring4D, I go back to use my TObjectList descendant, because in the end I figured out that all I need is just "generic collections with powerful features (loop, some, filter, etc) packed without bloating the EXE".

Hope it makes sense.

ab · 2021-12-21 07:32:13

The idea is to encapsulate the list within another regular class, if you need to add functionalities.
You use the list or the key/value for storage, then you add your logic on top of it.
This is what we did a lot with TSynDictionary or TDynArray in the framework.
My advice is to use composition rather than inheritance: use a fList: IList<> instead of inheriting from the list.

Otherwise, an IList<somerecord> or an IList<TsomeORM> could be useful to store some working process information, just as it is.
Behind the scene, it is just one (or two) dynamic arrays, which you access througout the methods.

edwinsn · 2021-12-21 08:36:15

Yes, "composition over inheritance"

ab · 2021-12-21 10:09:59

@dcoun
Please check https://github.com/synopse/mORMot2/commit/4a234d9a

dcoun · 2021-12-21 10:25:41

Now the object is added but when trying to access it with the last line in the code above the result for 0-index items is nil

ab · 2021-12-21 13:43:09

Try to debug why it is the case on your side because I can't reproduce it here.

Do the regression tests pass on your side?
There is a test for this exact case in https://github.com/synopse/mORMot2/commit/2758a7aa

dcoun · 2021-12-21 21:30:33

ab wrote:

Try to debug why it is the case on your side because I can't reproduce it here.
Do the regression tests pass on your side?
There is a test for this exact case in https://github.com/synopse/mORMot2/commit/2758a7aa

it works OK using the following and TobjectwithID instances:

v:=collections.NewList<TobjectwithID>([loCreateUniqueIndex]);

using the following type instances it does not

Ttest=class
  private
    fsoup:string;
  published
    property soup:string read fsoup write fsoup;
  end;

Should I register them before using them?

Last edited by dcoun (2021-12-21 21:31:15)

VojkoCendak · 2021-12-21 21:31:41

Hi, (D10.4.2 ent)

Is it possible to use IList<> for records ?
It reports Type is too complex.

thank you,
Vojko

dcoun · 2021-12-22 07:22:05

Adding the following to test.core.collections with the above Ttest type definition and it fails

  
  lt: ilist<ttest>;
  t:ttest;
.....
  lt := Collections.NewList<Ttest>;
  lt.Capacity := MAX + 1;
  for i := 0 to MAX do begin
    t:=ttest.Create; t.soup:=inttostr(i);
    Check(lt.Add(t) = i);
  end;
  for i := 0 to lo.Count - 1 do
    Check(lt[i].soup = inttostr(i));

I have noticed also that when adding in the following screen, value is none in add function
https://pasteboard.co/U3hiCtiA6j7N.png

Last edited by dcoun (2021-12-22 07:28:39)

ab · 2021-12-22 11:02:10

@dcount
You are right, TObjectWithID worked by chance.
I think I have fixed the issue about classes with https://github.com/synopse/mORMot2/commit/09e135f4

@VojkoCendak
The error message is explicit: you should use Collections.NewPlainList<> factory instead.
Also ensure you have the latest source code revision.

dcoun · 2021-12-22 12:33:57

Thank you very much Arnaud. It works now OK

VojkoCendak · 2021-12-22 14:09:17

@ab

Yes it works with plain records. (managed not yet );

thank you

VojkoCendak · 2023-04-18 07:15:06

Hi,

Is it possible to convert to loCreateUniqueIndex after mylist := Collections.NewList<TDateTime> creation ?
maybe: mylist.UniqeIndex := True;

thank you

Last edited by VojkoCendak (2023-04-18 07:15:37)

VojkoCendak · 2023-04-18 07:18:11

hi,

is it possible to:
1. copy/insert/add from another list in one line ?

thank you

ab · 2023-04-18 07:42:04

Hello,

1) no, the loCreateUniqueIndex option could be set only in the factory mehod, at creation.

2) yes, you can, accessing the TDynArray from the IList<T>.Data methods, so you can call TDynArray.AddDynArray().
I have added a new AddFrom() method to ensure we keep strong typing of the source list.
https://github.com/synopse/mORMot2/commit/8e541fb8

VojkoCendak · 2023-04-18 08:14:48

hi,

What about hash performance when it is copied from another list?

listnormal := Collections.NewList<TDateTime>;
listnormal gets populated ...
// want to copy from normal list to fast
listfast := Collections.NewList<TDateTime>([loCreateUniqueIndex]);
listfast.AddFrom(casi); // copy 
   // lots of iterations in some loop
    if listfast.IndexOf(cas)<0 then  // is slower when using AddFromversus just looping through listnormal and .Add
      anotherlist.add(cas);

Do we need to rehash it?

Last edited by VojkoCendak (2023-04-18 08:19:28)

VojkoCendak · 2023-04-18 08:26:16

yes,

I've added

  if Assigned(Another) then begin
    fDynArray.AddDynArray(Another.Data, Offset, Limit);
    if (fHasher <> nil) then
    	fHasher^.ForceReHash;
  end;

and it works

ab · 2023-04-18 13:56:06

My guess is that Hasher should NOT be handled as such because you may have some unexpected duplicates in ForceRehash.

Please try https://github.com/synopse/mORMot2/commit/51627ba2
(not fully tested yet)

VojkoCendak · 2023-04-18 14:15:04

I see, very nicely spotted.

thank you

mORMot Open Source

#1 2021-12-20 08:28:14

mORMot 2 Generics and Collections

#2 2021-12-20 10:26:21

Re: mORMot 2 Generics and Collections

#3 2021-12-20 12:10:48

Re: mORMot 2 Generics and Collections

#4 2021-12-20 13:34:32

Re: mORMot 2 Generics and Collections

#5 2021-12-20 14:18:14

Re: mORMot 2 Generics and Collections

#6 2021-12-20 14:34:24

Re: mORMot 2 Generics and Collections

#7 2021-12-20 14:47:59

Re: mORMot 2 Generics and Collections

#8 2021-12-20 15:09:02

Re: mORMot 2 Generics and Collections

#9 2021-12-20 15:46:08

Re: mORMot 2 Generics and Collections

#10 2021-12-20 16:14:37

Re: mORMot 2 Generics and Collections

#11 2021-12-20 21:00:23

Re: mORMot 2 Generics and Collections

#12 2021-12-21 07:05:02

Re: mORMot 2 Generics and Collections

#13 2021-12-21 07:32:13

Re: mORMot 2 Generics and Collections

#14 2021-12-21 08:36:15

Re: mORMot 2 Generics and Collections

#15 2021-12-21 10:09:59

Re: mORMot 2 Generics and Collections

#16 2021-12-21 10:25:41

Re: mORMot 2 Generics and Collections

#17 2021-12-21 13:43:09

Re: mORMot 2 Generics and Collections

#18 2021-12-21 21:30:33

Re: mORMot 2 Generics and Collections

#19 2021-12-21 21:31:41

Re: mORMot 2 Generics and Collections

#20 2021-12-22 07:22:05

Re: mORMot 2 Generics and Collections

#21 2021-12-22 11:02:10

Re: mORMot 2 Generics and Collections

#22 2021-12-22 12:33:57

Re: mORMot 2 Generics and Collections

#23 2021-12-22 14:09:17

Re: mORMot 2 Generics and Collections

#24 2023-04-18 07:15:06

Re: mORMot 2 Generics and Collections

#25 2023-04-18 07:18:11

Re: mORMot 2 Generics and Collections

#26 2023-04-18 07:42:04

Re: mORMot 2 Generics and Collections

#27 2023-04-18 08:14:48

Re: mORMot 2 Generics and Collections

#28 2023-04-18 08:26:16

Re: mORMot 2 Generics and Collections

#29 2023-04-18 13:56:06

Re: mORMot 2 Generics and Collections

#30 2023-04-18 14:15:04

Re: mORMot 2 Generics and Collections

Board footer