mORMot bigData future and actual state

gm · 2015-01-06 22:19:08

Hello,

This is my first post and firstly i would congrat mORMot team for this great software.

Well, I read recently about the mORMot bigdata future ( http://blog.synopse.info/post/2014/12/3 … is-BigData ).

It’s perfect timing and very interesting because I've been thinking about how I should design a web app with mORMot that takes advantage of amazon auto scaling and, of course, in that scenario the data is the challenge.

My first thought was DynamoDB, but I want avoid to depend on a proprietary infrastructure. Then I thought of mongoDB and that's when I found the post mentioned above.

It made me a little unsure about the actual state of mORMot scalability. My another option would be "ASP.Net/Azure" but I am inclined to mORMot because of their performance. However, I have some concerns and really appreciate if someone could give me some advice.

1) Does the mORMot has a significant better performance than asp.net when it work with no SQLite direct acess (mongoDB or external DB)?

2) MongoDB/TMongoClient -- Is it a good option in terms of performance? Why WiredTiger for the future?

3) How about the argue that the ORM layer by itself already degrades performance? This is true for the mORMot too? Should I consider programing SQL manually for best results?

4) What do you guys think is the best design in terms of scalability and high performance for a server app on the cloud?

ab · 2015-01-07 08:53:43

Welcome on this forum, and thanks for your interest!

1) About performance, I guess you want to compare to WCF+EF, the main differences are:
- performance has been a key element since the ground up: e.g. the whole mORMot stack avoids unnecessary memory move or allocation;
- more tunable threading abilities;
- our SynDB database access is definitively much faster than ADO.net, especially for individual reads;
- EF is slow, and AFAIK does not support NoSQL backends;
- EF is not client-server, whereas mORMot is a REST/JSON ORM;
- WCF is slower than mORMot written in Delphi due to garbage collection process, bigger memory use, immutable strings, "blackbox" approach, XML/SOAP roots (whereas mORMot is RESTful);
- Everything is integrated in mORMot, whereas with .Net, you will have to make several antagonist bricks talk together (e.g. WCF, EF, ado.net): for instance, the JSON transmitted by the server is emitted directly from the DB client layer, with not intermediary.

I used WCF services for years in my previous job: I can tell you than scaling is very difficult, and most scaling is to be done at DB level, or with vertical scaling.
Hosting in the cloud was not an option for this company, since the servers were so much resource demanding.
Configuring WCF is a nightmare (20% of our client support tickets were about .exe.config issues, or framework runtime problems, including breaking behaviors after a Windows update), whereas you can plug mORMot servers just by copying the executable and running it.

2) WiredTiger would be part of MongoDB 2.8.
We envisaged to access diretly WiredTiger in mORMot, but license restriction would probably make it not worth it (whole app should be GPL).

3) mORMot ORM is faster than plain SQL, since it relies on advanced techniques (like in-memory cache, global prepared statements, BATCH process with multi insert statements, direct export from DB to JSON, straight link to the REST layer), which are not available in other ORMs - in Delphi, Java or .Net.
For 95% CRUD operations, the ORM would be damn fast. See the numbers at http://synopse.info/files/html/Synopse% … ml#TITL_59 and compare with what you achieve with manual SQL, in ADO.Net or plain Delphi.
The TObjectList (static) lines refers to ORM process from in-memory, so you can see that the ORM layer is several orders of magnitude faster than a DB engine. mORMot's ORM would not be your application bottleneck.
In some cases, e.g. complex JOINed queries with aggregation functions, manual SQL may be faster: in this case, you can define SOA services - which is always a good idea, even to encapsulate ORM process. Consider using a DDD architecture - see http://synopse.info/files/html/Synopse% … ml#TITL_54

4) Horizontal scaling is the key.
But the fact that you can use less servers for the same number of concurrent clients is worth taking in account. mORMot high performance would help scaling vertically, before scaling horizontally. Vertical scaling helps for data consistency (e.g. for data cache).
Node.js is great, but JavaScript was not meant to write and maintain huge projects. You could use TypeScript or ObjectPascal (via SmartMobileStudio) to write your server code. But node.js has scaling issues, since it is asynchronous, whereas a mORMot server is multi-threaded, so makes a better use of the available CPU cores it runs on. And JavaScript, as a runtime, consumes a lot of memory, so is definitively slower than Delphi/FPC on high load.
For hosting, we are proud to start supporting Linux: hosting is cheaper, and performance is higher than Windows. Linux file system and memory management is usually faster than Windows, and a Linux server won't be overloaded by bloated GUI/AntiVirus/SlowUpdates which Windows suffers. We consider rather renting your own servers, run Linux on them, then copy your mORMot executable on it. It would be faster and much cheaper than any "cloud" solution, based on virtualization. And you will own your data.

ab · 2015-01-07 09:10:27

Ensure you read http://synopse.info/files/html/Synopse% … tml#SOURCE
About performance, some numbers for years ago (actual numbers are even better): http://synopse.info/forum/viewtopic.php?pid=6363#p6363

itSDS · 2015-01-07 09:39:23

Hi AB i read your concerns about WiredMonkey, are there any plans to change their GPL to LGPL ?
May be if they see what's going on with mongoDB and mORMot they may change it.

(Did you speak to them ?)

ab · 2015-01-07 12:30:07

@idSDS
No, I did not contact them.
And I should have to contact not the WiredMonkey guys, but their owner company, which is now the one owning MongoDB.
Honestly, there is no reason why they may change the license terms to please us.
Especially since MongoDB is already directly handled by MongoDB, at full speed.

gm · 2015-01-07 12:34:59

Thank you AB,

That helped me a lot. Just to clarify, I wasn't planning use EF. The only ORM I consider is mORMot due to performance.

About the horizontal scalability, that's exactly what I want but SQLite is not a good idea for the DB layer, right?

I'm looking for scalable and fast storage layer, even if I have to implement synchronization between the DB instances myself.

In your view, what is my best option?

ab · 2015-01-07 12:41:16

SQLite3 is the best idea about performance for the DB layer, up to some GB of data (knowing that SQlite3 consumes much less memory than other DBs).
Just check the numbers of our benchmark in our previous link.
Since SQLite3 is local to the server process, and very well integrated into mORMot, performance is amazing. Higher than any remote SQL or NoSQL DB.

For BigData, i.e. some TB of data, you would better use either a huge SQL DB (like PostgreSQL) or MongoDB (waiting for the upcoming 2.8 revision, which would feature WiredTiger).

gm · 2015-01-07 13:10:20

How about something around 100Gb? Do you think SQLite is a viable option?

Is that a good idea split the data into multiple files (by accounts for example)? Parallel connections to different files could be a performance problem?

Last edited by gm (2015-01-07 13:44:32)

ab · 2015-01-07 14:07:59

One SQLite3 database of 100 GB would still work pretty well if the needed indexes were created (just like with any DB), as reported by users http://stackoverflow.com/a/14454573/458259
Do you need 100 GB of SQL data? I guess no. This would be about BLOB probably.
If you want to store a lot of BLOB, a dedicated server (e.g. MongoDB) would make sense, e.g. accessed via SOA, or a dedicated ORM table via REST.
Then the main SQL data would be able to remain on the main SQLite3 DB.

One mORMot server instance per DB would make sense, of course.
If your data can be uncoupled, and has some size, it should benefit to be "sharded" into several mORMot servers.

Note that you can share the same single port between several mORMot HTTP servers, each one using its own "root" URI.
So it would be transparent for the user.
First implementation may use a single mORMot executable, then the internal mORMot servers, may later on be split into several mORMot executables.
You could also use TSQLRestServerRemoteDB to make redirection from a single mORMot server to a remote mORMot RESTful ORM on a diverse computer, if needed.

Data sharding would be enhanced later by the new feature requests, related to big Data.
But in its current case, you can already handle huge DB with mORMot.

Ensure you read the corresponding paragraphs in the documentation.
All those use cases are already documented.

gm · 2015-01-07 17:28:44

ab wrote:

...Do you need 100 GB of SQL data? I guess no. This would be about BLOB probably...

You're right, AB. Certainly I don't need of 100 GB of SQL data but let me explain:

My need is not about BLOB, my project is a server for a SaaS offer. Each service customer does not need 100 GB either, but I have to support hundreds of customer with 1 Gb needs and I intend a design that when a new VPS is launched by the load balance service, everything can be automated with no human intervention.

This is the real reason for my concerns about large data, however you helped me a lot.

Thank you so much!

edwinsn · 2015-01-08 03:41:56

@gm, I don't have done any test, but mORMot just feels fast than enough to me in my experiences.

gm · 2015-01-08 10:39:11

edwinsn wrote:

@gm, I don't have done any test, but mORMot just feels fast than enough to me in my experiences.

I know @edwinsn, I entirely agree with you. I'm just looking for the best approach for a scenario where maybe I can't have the HttpServer and the database in the same computer (VM) because of clones of VM httpServers are launched on demand automatically.

edwinsn · 2015-01-10 03:57:14

gm wrote:

I'm just looking for the best approach for a scenario where maybe I can't have the HttpServer and the database in the same computer (VM) because of clones of VM httpServers are launched on demand automatically.

You might consider PostgreSQL - I haven't used it, but from what I have read on Hacker News in recent years, PostgreSQL is getting better and better, two big things I remember are 1) full text search enhancement, and 2) enhanced support for JSONB (document-based storage, think MongoDB!)

And it'll be interested to see how you implement that load-balance-controlled booting of your web servers. Which cloud provider you are planning to use?

Last edited by edwinsn (2015-01-10 03:59:15)

ab · 2015-01-10 13:27:02

There is already a feature request about direct JSONB support - see http://synopse.info/fossil/tktview?name=ae27dd8032
We would like to implement direct and optimized support of JSONB columns for dynamic arrays, records or variant published records. At first, variant may be supported, since they are converted to/from JSON. Then SQL queries should be able to convert *DynArrayContains() functions - and eventually other more complex JSON queries - into corresponding JSONB operators.

But PostgreSQL replication is more complex to setup and maintain than MongoDB, as far as I discovered.

gm · 2015-01-13 11:14:11

Hi @edwinsn,

Thank you, It's really interesting this JSONB support on PostgreSQL but I am convinced that is the case where it's better a noSQL solution.

About "load-balance-controlled", I'm planning to use Amazon Elastic Load Balancing ( http://aws.amazon.com/elasticloadbalancing/?nc2=h_ls )

gm · 2015-01-13 13:10:47

@ab,

I'm thinking of to persist many TSQLRecord as a local file and as object on amazon S3. How does it seems to you? I'd appreciate it if you just put me in the right direction and point me the relevant info on documentation or website, especially on the objects serialization without the db layer.

ab · 2015-01-13 13:33:29

@gm
Are you considering using http://docwiki.embarcadero.com/Librarie … ageService ?
Perhaps TAmazonTableService does make better sense, if you want to store objects.

Amazon SimpleDB is for storing objects, whereas Amazon S3 is for storing BLOBs.
See http://aws.amazon.com/simpledb/details/

Amazon FAQ wrote:

Q: When should I use Amazon S3 vs. Amazon SimpleDB?
Amazon S3 stores raw data. Amazon SimpleDB takes your data as input and indexes all the attributes, enabling you to quickly query that data. Additionally, Amazon S3 and Amazon SimpleDB use different types of physical storage. Amazon S3 uses dense storage drives that are optimized for storing larger objects inexpensively. Amazon SimpleDB stores smaller bits of data and uses less dense drives that are optimized for data access speed.
In order to optimize your costs across AWS services, large objects or files should be stored in Amazon S3, while smaller data elements or file pointers (possibly to Amazon S3 objects) are best saved in Amazon SimpleDB.

I'm not convinced by the idea of paying Amazon, when you can host a mORMot RESTful server for almost nothing on a Linux box, and store the data using SQLite - or MongoDB if you want replication.

gm · 2015-01-13 17:37:06

ab wrote:

@gm
Are you considering using http://docwiki.embarcadero.com/Librarie … ageService ?
Perhaps TAmazonTableService does make better sense, if you want to store objects.

No, I didn't know about this Library (thank you). Actually, I'm thinking about store only blobs (json,html,etc) on S3, what I intend to do is minimise the amount of structured data, and then ease the overload on the database layer (SQLite or MongoDB).

ab wrote:

I'm not convinced by the idea of paying Amazon, when you can host a mORMot RESTful server for almost nothing on a Linux box, and store the data using SQLite - or MongoDB if you want replication.

That bothers me too, but by my estimates, seems to be cheaper than maintaining the necessary infrastructure for a horizontal scalability. Do you think I'm wrong?

Which Linux host provider do you recommend? In addition... "What environment (tools/IDE) would you recommend to work with mORMot on linux?" I have no experience with linux development.

gm · 2015-01-15 10:29:20

Does anyone have an opinion?

ab · 2015-01-15 12:37:09

Linux host provider depends on your country, and the countries where your clients will connect from.

About IDE under Linux, the only solution is to use Lazarus.
Debugging using Lazarus under Linux is pretty easy.
We added some information about how to create and install Lazarus in a Linux virtual box in our documentation - see http://blog.synopse.info/post/2015/01/1 … nks-to-FPC
The latest CodeTyphon 5.20 is an alternative, with some potential license issues, but was reported to work. See http://www.pilotlogic.com/sitejoom/inde … codetyphon

As an alternative, I'm currently considering using Kylix 3 from a Delphi 7 IDE - http://crosskylix.untergrund.net
In fact, the executable generated by Kylix is faster than FreePascal, and has no limitation about the RTTI, since it is the same front compiler than Delphi 7.
There is no debugger in CrossKylix, so logging is your friend here!
For application code, using FPC or Kylix won't change your code - but for the interface RTTI missing in FPC.
And Lazarus has a good debugger under Linux.

Note that Linux is a less stable platform than Windows for mORMot servers.
It is a platform we are currently working on.
And some features are not yet available.

gm · 2015-01-15 17:02:22

I found this: http://www.crossfpc.com/

they said:

Using an integrated toolchain, CrossFPC enables you to cross-compile your Windows® Delphi® applications to 32 bit and 64 bit native Linux applications for X86, Linux ARM and Android without ever leaving the IDE....
...This project was created for people who wish to develop cross-platform applications with Delphi while staying inside the familar Delphi IDE.

Does anybody knows about this project?

Last edited by gm (2015-01-15 18:52:18)

ab · 2015-01-15 18:55:03

CrossFPC is not worth it. Its compiler is outdated, and would not be able to compile mORMot source.
Lazarus is a breeze to install, and debugging within it, under Linux, is very good.
CrossFPC does not allow to debug, and it ships with locked binaries, whereas FPCUP or CodeTyphon will compile the whole FPC + Lazarus from the sources.

CrossFPC may be useful when cross-compiling to embedded devices (ARM), where no desktop would run.
But not for server applications like we need.

gm · 2015-01-16 13:21:17

Ok @ab, thanks for all help.

mORMot Open Source

#1 2015-01-06 22:19:08

mORMot bigData future and actual state

#2 2015-01-07 08:53:43

Re: mORMot bigData future and actual state

#3 2015-01-07 09:10:27

Re: mORMot bigData future and actual state

#4 2015-01-07 09:39:23

Re: mORMot bigData future and actual state

#5 2015-01-07 12:30:07

Re: mORMot bigData future and actual state

#6 2015-01-07 12:34:59

Re: mORMot bigData future and actual state

#7 2015-01-07 12:41:16

Re: mORMot bigData future and actual state

#8 2015-01-07 13:10:20

Re: mORMot bigData future and actual state

#9 2015-01-07 14:07:59

Re: mORMot bigData future and actual state

#10 2015-01-07 17:28:44

Re: mORMot bigData future and actual state

#11 2015-01-08 03:41:56

Re: mORMot bigData future and actual state

#12 2015-01-08 10:39:11

Re: mORMot bigData future and actual state

#13 2015-01-10 03:57:14

Re: mORMot bigData future and actual state

#14 2015-01-10 13:27:02

Re: mORMot bigData future and actual state

#15 2015-01-13 11:14:11

Re: mORMot bigData future and actual state

#16 2015-01-13 13:10:47

Re: mORMot bigData future and actual state

#17 2015-01-13 13:33:29

Re: mORMot bigData future and actual state

#18 2015-01-13 17:37:06

Re: mORMot bigData future and actual state

#19 2015-01-15 10:29:20

Re: mORMot bigData future and actual state

#20 2015-01-15 12:37:09

Re: mORMot bigData future and actual state

#21 2015-01-15 17:02:22

Re: mORMot bigData future and actual state

#22 2015-01-15 18:55:03

Re: mORMot bigData future and actual state

#23 2015-01-16 13:21:17

Re: mORMot bigData future and actual state

Board footer