You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 Re: mORMot 2 » How to Calculate Stock Quantity with Millions of Records with mORMot2? » 2024-09-13 00:57:26
Hi everyone,
I'm currently facing a performance issue in my application related to calculating stock quantities. The calculation involves summing up stock movements (in and out), and with millions of records and a lot of users accessing the data, the process is slow, causing the app to freeze and sometimes even result in deadlocks.
The usual architecture is to implement a job queue: The main app puts a job on the queue and immediately returns a "processing..." response to the client. Other worker apps take jobs from the queue, do the work, and communicate the outcomes by notifying the main app thru the queue or app database, notifying the client directly thru email/WhatsApp/Telegram/Discord etc.
But as ab wrote, if your database is the bottleneck, then another approach is needed, like running your DB on bigger iron or rethinking the database setup.
#2 Re: mORMot 2 » sqlite build fails on aarch64 » 2023-11-06 00:03:02
Another data point: All mORMot 2.1 tests pass on an aarch64-linux VPS running Ubuntu 22.04.
Software version tested: 2.1.5794 (2023-10-03 06:45:45)
Ubuntu 22.04.3 LTS - Linux 5.15.0-1040-oracle [utf8 5.7GB 50F0010]
1 x ARM Neoverse-N1 (aarch64)
...
Total assertions failed for all test suits: 0 / 77,402,651
#3 Re: mORMot 2 » mORMot with Azure Cosmos DB MongoDB » 2023-03-05 13:23:41
Very clear and interesting blog article!
I don't understand why the numbers are so low. I don't understand it is only the network latency.
With a local MongoDB instance, the test is run in a few seconds.
If you enable the logs in verbose mode, can you see something strange in the timing?
Thanks. I will try with logs. I think the poor performance could be because I'm using a free "teaser" Azure account. I have a paid account as well and will be continuing my experimentation using that account later on.
Perhaps you could add in the article that such a MongoDB instance is to be run from a VM server also hosted in the Azure Cloud - where the latency are likely to be "normal".
From a client/end-user place, using a cloud DB is not a common option, because it was not meant for this purpose.
More blog posts coming. :-) I will write about running mORMot in Azure in "cloud-native" style.
#4 mORMot 2 » mORMot with Azure Cosmos DB MongoDB » 2023-03-04 14:43:39
- PierceNg
- Replies: 2
Hi all,
I want to share that I have adapted the MongoDB example to work with Azure's MongoDB.
- Repo: https://github.com/PierceNg/mormot-cloud
- Blog post: https://samadhiweb.com/blog/2023.03.04. … azure.html
More to come...
Disclaimer: My blog has no monetization and I am not paid by MS or anybody to promote Azure.
#5 Re: mORMot 1 » Implementing OAuth2 » 2023-02-24 12:48:23
@Prometeus, I got your mail. Thank you. (My mail provider failed to deliver my reply mail.)
#6 Re: mORMot 1 » Implementing OAuth2 » 2023-02-20 06:06:38
@Prometeus, if you still have the gist content, please send me a copy by email. Thanks.
#7 Re: mORMot 2 » mvc-blog using Postgres - external content FTS4/5 table exception » 2023-02-16 12:36:41
FTS4/5 is a SQLite3 specific feature.
If you want to use another DB, you need to disable this feature.
Look at MVCModel.pas for {$ifdef/ifndef USEFIREBIRD
External database can't use FTS feature of SQLite, TOrmArticleSearch is not defined for USEFIREBIRD.
Thank you both. From @ab reply, I saw that and fixed that part accordingly. The app runs. But it fails to insert the fake articles. Blog info, author info, tags etc were inserted, but the articles weren't, due to a Postgres insertion exception. The relevant log entries are here: https://gist.github.com/PierceNg/4e4e24 … 19c8458385
The key line should be this:
20230216 12033029 ! EXC ESqlDBPostgres {Message:"TSqlDBPostgresStatement.ExecutePrepared:
Invalid array type ftBlob on bound parameter #10",
Statement:{Sql:"insert into public.Article (ID,CreatedAt,ModifiedAt,Title,Content_,ContentHtml,Author,AuthorName,PublishedMonth,abstract,Tags)Parameter #10 should be 'abstract'.
In Postgres, the columns show up as follows:
- id, bigint NOT NULL
- createdat, bigint
- modifiedat, bigint
- title, text
- content_, text
- contenthtml, integer
- author, bigint
- authorname, text
- publishedmonth, bigint
- abstract, text
- tags, bytea
#8 mORMot 2 » mvc-blog using Postgres - external content FTS4/5 table exception » 2023-02-16 10:03:49
- PierceNg
- Replies: 3
Hi,
I modified the mvc-blog example's MVCServerFirebirdIbx.dpr to use Postgres. Totally same code, only changing TsqlDBIbxConnectionProperties to TSqlDBPostgresConenctionProperties. The program runs - I can see the tables being created in the database - but then fails with following:
EModelException: TOrmArticleSearch is an external content FTS4/5 table but source Article is not a local SQLite3 table: FTS search will be unavailable
$00000000005A7BB1 InitializeTable, line 8940 of ../../src/orm/mormot.orm.core.pas
$00000000007C4570 CreateMissingTables, line 1603 of ../../src/orm/mormot.orm.sqlite3.pas
$0000000000402A8C $main, line 54 of MVCServerPostgreSQL.lprWhat should I look at to fix?
Program built with Lazarus 2.2.4 with FPC 3.3.1 on Ubuntu 20.04. Postgres is official Postgres 15-Alpine Docker image running on the same host.
The standalone SQLite server works.
#9 Re: mORMot 2 » Error parsing .static files » 2023-02-01 09:14:49
Thank you. Tested ok.
#10 Re: mORMot 2 » Error parsing .static files » 2023-02-01 00:47:00
mORMot newbie here, just started exploring the code base. I ran into this yesterday and thought it was intended behaviour. Ended up writing a simple method-based service to serve static content that runs together with the MVC app.
Pages: 1