mORMot Open Source

zhangguichao

Member

Registered: 2023-03-21

Posts: 29

Ask questions

Are there actual cases of big data volume and concurrency?

I am very interested in this framework, usually use mssql, I don't know if I can switch to mormot+sqlite

Thanks for  answer ?

Last edited by zhangguichao (2023-04-18 09:06:09)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Ask questions

If you want to scale, don't think in terms of one centralized database.

In a former job, I have used mORMot to create a mesh of 500 barebone servers each with its own SQLite3 + files storage.
It was meant to scale to 1 PetaByte of data (each server having a 2TB harddrive). Using SQLite3 for metadata, and local files for the huge dataset storage.
And it worked: we were able to make a distributed RegEx on log files with a speed of 1TB/s.

About "pure SQlite3" databases, I have seen some on production of more than a few dozen GB with no problem. Ensure you create the proper indexes, and used a page size of 4096 bytes when initializing the DB.
But the main idea is to shard the data into several SQlite3 databases, if you can - e.g. one SQLite3 file per customer or per group of customer.
It is safer, easier to backup/duplicate/move, and much better for scaling.
Creating a SQlite3 database file is very easy - much easier than creating a regular SQL DB.

zhangguichao

Member

Registered: 2023-03-21

Posts: 29

Re: Ask questions

Thank you very much for your answer

Last edited by zhangguichao (2023-04-19 00:56:24)

zakaria

Member

Registered: 2021-06-21

Posts: 26

Re: Ask questions

Hi ab,
depends on your answer at #2,
* how many mormot services do i have to scale those sqlite dbs, or it would be one with some techniques to separate them ?!
* what do you mean of pageofsize and propper indexes, has mormot create/managed them by default ?
Thanks

dcoun

Member

From: Crete, Greece

Registered: 2020-02-18

Posts: 430

Re: Ask questions

This a really interesting proposal @ab
As I can understand, you propose to divide parts of an application to different TrestserverDBs (each one with an SQLITE database) if high level processing does not need to access more than one sqlite's tables for its tasks. That will even allow not to need an external database and simplifies backup and other tasks.
The problem is that for special cases, an sql query needs to combine(join) and have read access to more than one sqlite's tables. SQLITE provides an "attach" feature that allows different sqlite databases' tables to be accessible from an other sqlite db as its tables.
Can a TrestserverDB have an sqlite db with attached tables from other TrestserverDB's sqlite? how reliable is that and what is needed for functioning without troubles? That will be an interesting scenario and example.

Last edited by dcoun (2023-04-24 08:16:30)