Need general advises for a lot of small databases & cached records.

edwinsn · 2020-11-08 18:03:52

I have a project might be a little different than the usual use cases, and I want some general advises as to reducing **memory use** and **improving performance**.

- The program will open a folder which might contain a lot of files and subfolders.
- In each subfolder there is a sqlite db and its corresponding TSQLRestServerDB be created.
- for each file the program will create an object of a TSQLRecord-decendent for it.

The problem is that for folders with many subfolders, the RAM consumption go up.

Any options to reduce the memory consumption for TSQLRestServerDB and TSQLRecord objects?

Thanks!

ab · 2020-11-08 20:12:58

What takes memory is the SQLite3 internal cache.

You can
1. Reduce the cache size assigned with each TSQLRestServerDB
2. Maintain a pool of TSQLRestServerDB, and keep only e.g. the last 20 instances in memory

I usually used a combination of the two, on production, in such use cases.

edwinsn · 2020-11-09 08:54:46

@ab,

Thanks! Very helpful!

I also noticed TSQLDataBase.SetUseCache will create a TSynCache object:

TSynCache.Create(16384*1024,true)

which is a 16-Mb framework-level cache.

Unfortunately, the size of the TSynCache size is hard-coded. Any chance to make it configure-able?

ab · 2020-11-09 09:23:17

Please check https://synopse.info/fossil/info/26dc0a3497

edwinsn · 2020-11-09 10:38:25

Great! Thanks! Excellent support of an open source project, as always!

mORMot Open Source

#1 2020-11-08 18:03:52

Need general advises for a lot of small databases & cached records.

#2 2020-11-08 20:12:58

Re: Need general advises for a lot of small databases & cached records.

#3 2020-11-09 08:54:46

Re: Need general advises for a lot of small databases & cached records.

#4 2020-11-09 09:23:17

Re: Need general advises for a lot of small databases & cached records.

#5 2020-11-09 10:38:25

Re: Need general advises for a lot of small databases & cached records.

Board footer