Posts by TPrami

TPrami · mORMot 2

ab wrote:

AFAIR the ORM is already using UNNEST for both INSERT and UPDATE - and also DELETE with a similar ANY() trick.
Since years.

Nice!

TPrami · mORMot 2

Yo,

dunno if this is in use, or relevant to mORMot, but take or leave it

https://www.timescale.com/blog/boosting … rformance/

-Tee-

TPrami · mORMot 2

Could also help if you'd shared, what kind of problems you encountered.

-tee-

TPrami · mORMot 2

ab wrote:

But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.

That is worrying indeed, hope he is OK.

-Tee-

TPrami · mORMot 2

ab wrote:

With no stack trace it is very difficult to investigate...

Any luck in finding the bug?

-Tee-

TPrami · mORMot 2

mpv wrote:

PR 9121 is ready. If everything goes well, I'll bring '-O4' back in next PR - it makes tiny improvements

Please post link to results when available.

This is more interesting to follow than most movies on Netflix

TPrami · mORMot 2

Hello,

Why that would have to be part of mORMot, seems like Business logic, not necessarily beneficial to the framework it self...

-Tee-

TPrami · Low level and performance

ab wrote:

15M/s = 15 millions of hashes per seconds = per call speed
3GB/s = 33 GigaBytes per second = per data size speed

Ah, makes sense, thanks.

-Tee-

TPrami · Low level and performance

Hard to tell are those parallel or non-parallel versions. What they publish.

SSE2 and New AVX (what ever it was) optimizations give quite nice boost.

2500 crc32c in 157us i.e. 15.1M/s or 33 GB/s

Don't understand those values, 15M/s or 3GB/s, would need header for table/and table

-Tee-

TPrami · Low level and performance

https://xxhash.com/doc/v0.8.2/index.html

-Tee-

TPrami · mORMot 2

Uefi wrote:

Is it even possible to make a BloomFilter without false positives?

As far as I know whole point of BloomFilter is that it allows False Positives. It is designed that way. And should be used in cases where that is OK.

-Tee-

TPrami · mORMot 2

Uefi wrote:

Because the files that I process are very large 32-100 gigabytes, now imagine how much RAM the same THashSet will steal ?
By smaller standards, THashSet steals three times more RAM than the file itself 32*3 or 100*3 = 100-300 gigabytes RAM

Seems to be valid point. IF the processed file is tight (not and XML or JSON file, but binary etc), most likely it is too much to keep in memory anyways.

Depending what you are actually doing it could be better off in DB, or if it is some kind of lookup, wiit some bias/pattern maybe can have active HOT cache and if not found check from DB etc...

hard to tell, these are just shots in the dark...

-Tee-

TPrami · mORMot 2

Uefi wrote:

Can you please tell me how I can achieve 0% collisions without taking up too much RAM?

Why are you so worried about RAM?

Most computers have gigabytes of free RAM, and it is pretty cheap to get more. Your mileage might vary, it all depends on everything for sure.

-Tee-

TPrami · mORMot 2

Hello,

I've been wondering how close mORMot itself is in the TFB compared to Dev version.

Is it synced all the time or feature by feature. Or is some functionality and changes developed over there and merged back etc...

-Tee-

TPrami · mORMot 2

ab wrote:

@pavel Hope you are not too bad. We think daily about you, your family and your country.

True, hope everything is as OK as it just can be...

TPrami · mORMot 2

ab wrote:

A Delphi 12 hotfix seems on its way

Time just crawls for the one who waits for the Delphi HotFix.
- Possibly Albert Einstein

-Tee-

TPrami · mORMot 2

ab wrote:

@TPrami
Sad to hear, but interesting.
Do you have any references about those D12 bugs?
Are they about the IDE (instability, incompatibility), or the compiler itself?

All over. Worst are the compiler problems.

new div optimization does not always work and maybe there was something in Mod also (Can't remember). IDE has broken features.

I think I saw following, could not reproduce after changed the code.
function Foo: Ingeger; overload;
function Foo(const AValue: Integer): Integer; overload;

Then I called Foo;, It was kind of Foo, but wirth 1 as default parameter. Then Called witrh Foo(0); and at least debugger went to the Foo; (without parameter). Made simple app but did not behave same way.

Collected few highly voted tickets.

https://quality.embarcadero.com/browse/RSP-43580
https://quality.embarcadero.com/browse/RSP-43424
https://quality.embarcadero.com/browse/RSP-43540
https://quality.embarcadero.com/browse/RSP-43485
https://quality.embarcadero.com/browse/RSP-43586
https://quality.embarcadero.com/browse/RSP-43445
https://quality.embarcadero.com/browse/RSP-43418
https://quality.embarcadero.com/browse/RSP-42692
https://quality.embarcadero.com/browse/RSP-43274

-Tee-

TPrami · mORMot 2

tech wrote:

Hi all,
I've tested the mormot v2.2 under Delphi12 and windows 10 and I got some failing regression tests :

D12 has some nasty bugs, so there might be quite random stuff happening when running any code compiled with it.

Eagerly waiting for the update to it. Hope they really take their time tough, and squish all bugs they just can.

-Tee-

TPrami · Low level and performance

ab wrote:

I am dubious this article is very accurate.
Speaking of 'integer division' for a division per a constant on a recent compiler, in the context of performance is wrong.
Any good compiler (gcc, clang, FPC but not Delphi) is able to replace a " x / 10 " with a multiplication with the 32-bit or 64-bit reciprocal.
I do not see how its implementation may be faster than our StrUInt32() or StrUInt64() asm for i386 or x86_64.
In the context of pascal, the real bottleneck of IntToStr() is not the text computation, it is the string allocation.

True.

Don't know how that algorithm would perform, I would say just by reading the article and looking at Delphi implementation.

My gut feeling is that the speed up promises of that article are way too large. Donät understand C/C++ enough that I could port that code to Delphi/Object Pascal.

-Tee-

TPrami · Low level and performance

https://towardsdatascience.com/34-faste … 2453d25352

-Tee-

TPrami · mORMot 2

ab wrote:

There are continuous 24/7 rounds, but only one "official" round each year.

Roger that...

-Tee-

TPrami · mORMot 2

ab wrote:

I suspect next official round 23 will be next Summer...

I thought that it'll run like couple times a week or something.

But in any case. Follow mORMot progress in this eagerly.

-Tee-

TPrami · mORMot 2

ab wrote:

We made a lot of tuning after participating to the TFB challenge.
https://www.techempower.com/benchmarks/ … =composite
We finished #12 so it is pretty good.

OT but...

Is TFB challenge is over or mORMot is not participating anymore?

-Tee-

TPrami · Delphi

Couple of nasty bugs in D12 compiler. Please vote:

https://quality.embarcadero.com/browse/RSP-43274
https://quality.embarcadero.com/browse/RSP-43418

(if previously discussed in here, don't read the message ? )

-Tee-

TPrami · Delphi

If they would update PCRE-engine to latest

https://quality.embarcadero.com/browse/RSP-42524

TPrami · Low level and performance

Yellow,

Never pondered what happen when memory gets fragmented. Bit hard to test maybe. But if there would be easy way, could give good info of performance when environment is in challenging state.

https://johnnysswlab.com/faster-hash-ma … ification/

-Tee-

TPrami · mORMot 1

Also found this:

https://github.com/DenisAnisimov/ZSTD.pas

There is also LZ4 implementation.

Have no idea of the quality tough.

Both are interesting compression algorithms for sure.

TPrami · mORMot 2

Relax...

TPrami · mORMot 1

ab wrote:

$215/month * 2 it is a lot of money just for testing

Let me check my Lottery ticket.... Wait a sec...

Nope, can't sponsor right now...

-Tee-

TPrami · mORMot 1

Depending on how that cloud works, that would be quite cheap way to get tested on high end machine.

IOPS one has pretty impressive stats.
i1-180 180 GB 32 50 GB SSD 4 x 1.9 TB 8 Gbps guaranteed 4 Gbps max. $1.96 /hour

If could remotely start, update things, and run, get results and show down the machine.

Beast would be if there could be pre paid credit, put like 200€ and if the server would stop working after it is used, then it would be safe, if some mishap.

Bet there is some cloud operator who has some testing frendly plan. Which would prevent ridiculously expensive mishaps if something keeps on running there, even one things it is shut down.

-Tee-

TPrami · mORMot 1

ab wrote:

And from my tests, libdeflate at level 1 (what we use for HTTP) is faster than brotli.
I will include the latest libdeflate 1.13 statics in the next days, which made level 1 even faster than before.
This AWS blog article is not very accurate.
They just missed the fastest zlib library around, which is libdeflate. Much faster than cloudflare fork, because it is a full rewrite.

Slightly off topic but made qp fro delphi for the ZLib implementation, if someone cares, please vote and give more info: https://quality.embarcadero.com/browse/RSP-38978

-Tee-

TPrami · mORMot 1

IF ZLib is used, happened to stumble upon this:

https://aws.amazon.com/blogs/opensource … lib-forks/

Quite a bit faster forks of ZLIb, some not API compatible, and some only for in memory operations, but anyhow. check those out.

-Tee-

TPrami · PDF Engine

Is code/tool still available somewhere?

All links seem to point something obsolete.

-tee-

TPrami · mORMot 1

OK,

I could do somethiong like that.

if code is something like that

whilke Data do
Compress(GetFewBytesOfData);

make the Compress() keep cache internally which is flushed at end and when buffer gets full.

I take a look at the file compression routine.

Thanks man.

-Tee-

TPrami · mORMot 1

Yo!

Did not find with quick search (or did not understand what I saw).

So would it possible to use SynLZCompress1() to compress data in lets say 10kb chunks instead of whole data at once.

I have use case where I would like to avoid to have single large piece of the data in memory all at once (op there is possibility of that).

To be clear I mean that I would like to get same result with chunks than as I would if I had all the data in one large buffer. So the data would be fed in same pipeline but not in one go. (So not just small individual pieces of compressed data, but same binary data as if I would compress in one go, hope someone will understand this )

-Tee-

TPrami · mORMot 1

I was thinking also that mORMot2 would be way to go

TPrami · mORMot 1

Basically I do not care who takes the project when/if it comes alive

I just posted here if some company would reach back to me for it. Would be very pleased if it based off of mORMot.

Just offering (preminilary) business opportunity, to mORMot community.

Hope this makes it bit more clear.

Also trying to point out if this project goes live, I would try to influence that as much of it as possible is released as Open Source so community can benefit, if something along those lines is developed.

-Tee-

TPrami · mORMot 1

Hello,

First of all, Sorry that I can't give much details on this.

One Finnish nonprofit organization is starting project that would need an server. As far as I can tell (so far), server would be pretty trivial. It'll would be for Android and iOS client. Would be Windows or linux server, not sure yet.

If everything goes as I would like it to, if needed to develop libraries or mORMot, all paid work, would be released as open source, but not the final product.

I would need some very rough price per some unit (which your company uses).

And even better if you can make single source Android and iOS app even better, only difficult on these apps is that it should be able to take photos and have some ML/AI thingy that tries to detect object. (Objects are similar, but might be different shape or color, should give some probability of detection )

Feel free to ask, not sure can I give specific answer tough

Sorry for quick and dirty post.

-Tee-

TPrami · Delphi

Problem very poorly explained at : https://quality.embarcadero.com/browse/RSP-21814

Please vote for it.

Now we have seen many cases of this that customers have bought new computer and this setting has been on as default. And then at least our software stops working.

Hope that you'll test this in on your own environments and products. and report your findings on that Ticket. I'll try to get on this also day, and report more exact findings.

-Tee-

TPrami · mORMot 1

ab wrote:

I am preparing the new repository, and start naming things and folders.
Check https://github.com/synopse/mORMot2
We propose to switch to dotted named units, e.g. mormot.core.json.pas

WOOOHOO!!!!

TPrami · mORMot 1

mpv wrote:

Nice!
BTW size of SynCommons is not a problem for Lazarus, only for Delphi.

Size of problem for Programmers mainy, easy to start new project, couple of units in uses and that's it. But pain to go through, be SyCommons have lot of stuff most of users have not even seen

I would prefer unit scopes/namespaces but that would mean dropping quite many delphi versions (dotted filenames) mORMot.JSON.Utils /mORMot.UF8.Utils ... (or similar)

For older delphi versions there would always be 1.18, so not the worst thing to drop all but pretty much patest delphi versions.

-Tee-

TPrami · Low level and performance

thanks for info`?

TPrami · Low level and performance

Has it been measured yet? Memory usage/Speed etc? Is there clear measurable menefit?

-Tee-

TPrami · Delphi

Sorry, posted wrong link like 4 times

-tee-

TPrami · Delphi

With haters unite, please vote for this :

https://quality.embarcadero.com/browse/RSP-18953

Compiler (etc) assisted with removal Refactoring tool would be super cool.

Sometimes so error prone to remove by hand, especially nested withs.

(oops wrong link at start)

TPrami · Language

That was the case, which is kind of bummer.

Went with invoking the Setter-method to update the value. Need to add

{$RTTI EXPLICIT
METHODS([vcPublic, vcPublished, vcProtected])
PROPERTIES([vcPublic, vcPublished, vcProtected])
}

and use this code to kick Setter:

class procedure TInterfaceMethodInvoker.InvokeMethod<T>(const AIntf: IInterface; const AMathodName: string; const AValue: T);
var
LObj: TObject;
LRttiContext: TRttiContext;
LRttiType: TRttiType;
LRttiMethod: TRttiMethod;
LValue: TValue;
begin
LObj := AIntf as TObject;
LRttiContext := TRttiContext.Create;
try
LRttiType := LRttiContext.GetType(LObj.ClassInfo);
LRttiMethod := LRttiType.GetMethod(AMathodName);
LValue := TValue.From<T>(AValue);
LRttiMethod.Invoke(LObj, [LValue])
finally
LRttiContext.Free;
end;
end;

TPrami · Language

I've got few interfaces I need to update, they are not inherited from common ancestor (Would have been too easy). They have properties with same name, and datatype.

So to save tons of repeating code I could use RTTI to update the properties by name. Sadly did not find any good example on RTTI and Interface properties.

If someone has code to donate or point me into the right direction, I would be very happy camper...

-Tee-

TPrami · mORMot 1

macfly wrote:

But... Which programmer likes to talk, interact socially?
- I'm just joking, guys
I agree with your idea of creating a repository where everyone can test, learn, and teach.

I was thinking the same, for a couple of months, then I thought there is bound to be one that is exception to the rule.

-Tee-

TPrami · mORMot 1

At this stage it would be too much of an question, I could open up the repository, maybe, and then someone could check it out.

But it would not be interactive way I would I'd liked it to be. Also I think it would be fun to chat with fellow programmer from somewhere else anyways.

-Tee-

TPrami · mORMot 1

Yellow,

I'd like to do something completely different at home and I made an start with mORMot some time back, but did abandon that hobby project. Then I basically took that and started new one, but like the first time have serious doubts are my design choices OK. and mainly are they sane in perspective of mORMot.

If someone is willing to sacrifice some of their free time to chat with fellow coder and check my mORMot doodles out, would be more than happy.

I understand that most of you would not like to do it free, payments would most likely be bigger hassle than the money I could spare, for most. Not saying there would not be any kind of compensation but not as default, at least now.

Please don't bash me about this message, I've been thinking of sending this message like month or two now, so it is not easy to ask for help either

-Tee-

mORMot Open Source

#1 Re: mORMot 2 » Improving PostgreSQl insert speed » 2024-11-19 10:12:46

#2 mORMot 2 » Improving PostgreSQl insert speed » 2024-11-18 08:30:06

#3 Re: mORMot 2 » Delphi Package (.dkp file) » 2024-11-13 05:30:35

#4 Re: mORMot 2 » High-Performance Frameworks » 2024-09-16 09:58:08

#5 Re: mORMot 2 » High-Performance Frameworks » 2024-09-16 07:24:58

#6 Re: mORMot 2 » High-Performance Frameworks » 2024-06-18 08:14:21

#7 Re: mORMot 2 » Cryptocurrency in mORMot ? » 2024-05-28 09:30:39

#8 Re: Low level and performance » XXH3_64bits() really flies » 2024-05-17 08:11:39

#9 Re: Low level and performance » XXH3_64bits() really flies » 2024-05-16 10:40:00

#10 Low level and performance » XXH3_64bits() really flies » 2024-05-16 07:51:01

#11 Re: mORMot 2 » SynBloomFilter doesn't work correctly » 2024-04-24 05:09:46

#12 Re: mORMot 2 » SynBloomFilter doesn't work correctly » 2024-04-23 04:45:42

#13 Re: mORMot 2 » SynBloomFilter doesn't work correctly » 2024-04-23 04:03:52

#14 Re: mORMot 2 » High-Performance Frameworks » 2024-04-10 03:43:41

#15 Re: mORMot 2 » High-Performance Frameworks » 2024-01-15 05:54:49

#16 Re: mORMot 2 » mORMot2 Regression Tests fails » 2024-01-06 07:10:17

#17 Re: mORMot 2 » mORMot2 Regression Tests fails » 2024-01-05 11:21:21

#18 Re: mORMot 2 » mORMot2 Regression Tests fails » 2024-01-05 04:40:11

#19 Re: Low level and performance » IntrToStr algorithm (Interesting read) » 2023-12-08 08:47:43

#20 Low level and performance » IntrToStr algorithm (Interesting read) » 2023-12-07 05:44:44

#21 Re: mORMot 2 » Great performance framework » 2023-11-29 08:28:14

#22 Re: mORMot 2 » Great performance framework » 2023-11-28 18:16:09

#23 Re: mORMot 2 » Great performance framework » 2023-11-28 15:19:35

#24 Delphi » Couple of nasty bugs in D12 compiler » 2023-11-27 04:48:32

#25 Delphi » Made feature request for Embarcadero to update PCRE, vote if care... » 2023-10-31 08:51:14

#26 Low level and performance » An interesting read on implementing some data structures » 2023-10-03 06:17:01

#27 Re: mORMot 1 » ZSTD support » 2023-09-13 06:05:58

#28 Re: mORMot 2 » Away From Keyboard for 2 Weeks » 2023-05-22 08:40:48

#29 Re: mORMot 1 » High-performance frameworks » 2023-01-03 08:02:57

#30 Re: mORMot 1 » High-performance frameworks » 2023-01-02 08:52:04

#31 Re: mORMot 1 » High-performance frameworks » 2022-08-30 04:05:17

#32 Re: mORMot 1 » High-performance frameworks » 2022-08-23 04:48:37

#33 Re: PDF Engine » htm2pdf free tool release » 2022-05-27 05:48:38

#34 Re: mORMot 1 » "chunked/stream" compression with SynLZCompress1() (etc) » 2022-02-19 18:11:13

#35 mORMot 1 » "chunked/stream" compression with SynLZCompress1() (etc) » 2022-02-18 14:29:43

#36 Re: mORMot 1 » [Slightly Offtopic] Preliminary query of company to implement server » 2020-11-18 09:28:35

#37 Re: mORMot 1 » [Slightly Offtopic] Preliminary query of company to implement server » 2020-11-18 09:10:28

#38 mORMot 1 » [Slightly Offtopic] Preliminary query of company to implement server » 2020-11-18 06:13:20

#39 Delphi » Test your product with UTF-8 beta setting » 2020-03-19 06:10:21

#40 Re: mORMot 1 » Revision 2.x of the framework » 2020-03-12 11:28:11

#41 Re: mORMot 1 » Revision 2.x of the framework » 2020-03-12 09:42:58

#42 Re: Low level and performance » What or are benefits of using Daniel Lemire's method for Dicti » 2020-02-18 05:23:00

#43 Low level and performance » What or are benefits of using Daniel Lemire's method for Dicti » 2020-02-17 12:51:12

#44 Re: Delphi » With haters unite » 2020-02-06 11:16:59

#45 Delphi » With haters unite » 2020-02-06 07:43:14

#46 Re: Language » Does anyone know how to update value of Interface property with RTTI » 2020-01-16 08:22:40

#47 Language » Does anyone know how to update value of Interface property with RTTI » 2020-01-10 05:59:53

#48 Re: mORMot 1 » Anyone up to doing some mORMot tutoring (for me)? » 2019-06-06 05:38:32

#49 Re: mORMot 1 » Anyone up to doing some mORMot tutoring (for me)? » 2019-06-05 06:35:08

#50 mORMot 1 » Anyone up to doing some mORMot tutoring (for me)? » 2019-06-04 07:07:34

Board footer