the lack of scale solution

keinn · 2018-10-27 02:52:20

for example, our online services has many Couppon infomation , currently we use the ORM Rest Part of mORMot to serve the requests from all kinds of clients, such as browser ajax, mobile apps , pc clients.
something like this:

  TSQLCoupponInfo = class(TSQLRecord)
  private
    fId: TID;
    fProductName: RawUTF8;
    fManufactor: RawUTF8;
    fPrice:Currency;
    fCouppon:Currency;
    fDueTime:TDateTime;
    fCreatedAt: TCreateTime;
    fModifiedAt: TModTime;
   etc...
    
  published
    property id:TID read fId;
    property ProductName :rawuft8 read fProductName write ProductName;
    etc ...
    property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt;
    property ModifiedAt: TModTime read fModifiedAt write fModifiedAt;

  end;

many table like this contains more then 20 fileds,and such table row count growing so fast (nearly 200 thousand per day),
now it contains nearly 5million rows .
the valid Couppon info is just the recent few days, we do not need to lookup the whole table,
but mORMot does not have a way to do something like :generate one table per day （or other strategy）

by this project, we build it all the mORMot rest way, no stored proc, few table relation etc.
now we facing the rapid growing table size and massive client request（nearly 1million anonymous UV, 10million PV） , and our marketing invest will make the load way more heavy in the foreseeable future. not just the public services, many other services need auth also faceing the heavy requests.

now only thing we do ,it's just put many CDN server at Services's frontend and use more powerful machine.
we go through all the mORMot doc and forum info, we just found AB said that :you dont need scale or loadbalancing.

but we are so afraid that one day , our services just can't hold the request load anymore.
so, we need such sugestion:
1. table scale to control single table row count;
2. Rest Services load balancing.
and another thing , rest services really need version control .

we only use delphi to build or whole project, only 3 coder , and we love delphi/pascal , we really dont wanna switch to Java or Go solution.
maybe we used mORMot the wrong way . but any advice will be appreciated.
thanks very much.

Last edited by keinn (2018-10-27 02:59:40)

edwinsn · 2018-10-27 03:54:49

keinn wrote:

1. table scale to control single table row count (the valid Couppon info is just the recent few days, we do not need to lookup the whole table);
2. Rest Services load balancing.
3. rest services really need version control .
4. we only use delphi to build or whole project, only 3 coder , and we love delphi/pascal , we really dont wanna switch to Java or Go solution. maybe we used mORMot the wrong way . but any advice will be appreciated.

1. Did you create index for CreatedAt and ModifiedAt? I believe the database system can take care of the indexing and allow you query the records by the datetime efficiently.

2. I believe ab is not against the load balancing function, the current company he works for has several hundreds of server running mORMot-based service. Clue 1: http://blog.synopse.info/post/2016/07/3 … d-Big-Data

3. I don't know how other languages do the API versioning, but with mORMot you can version the API with interfaces or function names, eg. IMyInterfaceV1, IMyInterfaceV2...

4. You don't have to, maybe other languages offer more ready solutions for load balancing, but I believe Delphi/Pascal with mORMot is flexible enough to allow you to program solutions for the issues. Please take a read of this post (https://synopse.info/forum/viewtopic.php?id=4703), ab's company will have over one thousand servers running, it's based on a so-called **queue-based** architecture, so does the the OP's architecture.

And let's see other's suggestions.

ab · 2018-10-27 07:59:55

As edwinsn wrote, a SQlite3 table with millions of rows is not a problem to query, if you lookup by the primary key (TSQLRecord.ID) or proper indexed field(s).
And if you want to query by time/date periods, a TSQLRTree index may be worth it - see https://sqlite.org/rtree.html

First you may just use a HTTP reverse proxy load balancer, using the URI to balance to several mORMot servers.
Your services URI should allow a stateless per-convention distribution, e.g. define group of users, so that a given user will always access the same mORMot server.

For any scaling project, we never expose the ORM over REST directly, but use several SOA interface-based services.
Internally, the SOA services may use more than one ORM database.
It is pretty easy to create a new SQLite3 database file e.g. per family/group of users, or per period of time.
The logic has to be implemented in your SOA service itself. We use a TSynDictionary for maintaining a list of database processing classes, each one with its own data folder, and with a timeout parameter to release the unused resources after a few minutes.
Then you can put a public mORMot API SOA as front-end, to do the load balancing in pure object pascal code.

My guess is that your project architecture is still too much database centric.
You seem to refer to the database as the center of your process.
Try to switch your architecture to a mesh of microservices (which may be stand-alone or within a single process), each one with its own persistence, i.e. with its own SQlite3 db.

edwinsn · 2018-10-31 06:07:04

I came across the Google Cloud Tasks introduction blog post today, check it for ideas: https://cloud.google.com/blog/products/ … n-runtimes

with mORMot we can definitely implement a similar queue-based architecture!

mORMot Open Source

#1 2018-10-27 02:52:20

the lack of scale solution

#2 2018-10-27 03:54:49

Re: the lack of scale solution

#3 2018-10-27 07:59:55

Re: the lack of scale solution

#4 2018-10-31 06:07:04

Re: the lack of scale solution

Board footer