You are not logged in.
We almost finished implementing a long-standing feature request, introducing MVC / MVVM for Web Applications (like RubyOnRails) in mORMot.
This is a huge step forward, opening new perspectives not only to our framework, but for the Delphi community.
The mORMot point of view is unique, and quite powerful, since it is integrated with other parts of our framework, as its ORM/ODM or interface-based services.
Of course, this is a work in progress, so you are welcome to put your feedback, patches or new features!
This is the forum thread for http://blog.synopse.info?post/2014/10/2 … App-mORMot
See http://blog.synopse.info/public/mORMot/mORMotMVC.pdf for the mORMot MVC Draft Documentation!
Offline
Sample "30 - MVC Server" will now handle tags (categories) for articles.
Tags are stored as a dynamic array of integers within TArticle.Tags, without any pivot table.
They are displayed as text, in alphabetic order.
Performance of the BLOG engine is awesome: if you create 200,000 fake blog articles and 400,000 comments (written in less than 20 seconds in the SQLite3 DB - just define FAKEDATA_ARTICLESCOUNT=200000 in MVCViewModel.pas and erase the previous .db), the web application works as fast as the light, including tag search, and simple paging. The DB contains more than 340 MB of text...
WordPress administrators would start to cry, I suppose...
Offline
@ab,
R 1.18.407
What happened to the Category table in MVCServer.db?
Michal
Add:
It is okay.
It SynDBExplorer remembered the old structure.
(What is not OK).
Michal
Last edited by miab3 (2014-10-25 20:15:42)
Offline
The TSQLCategory has been replaced by a TSQLTag table, and the tags are stored as dynamic array within TSQLArticle.Tags.
Welcome to the "data sharding" world.
I'm convinced we can write a whole BLOG server without any JOINed query.
Offline
@ab
There is something wrong with the dates(years) in the Archives list.
R 1.18.410
Michal
Last edited by miab3 (2014-10-27 09:08:13)
Offline
great work ab!
do you think is possible the same structure for desktop application?
Offline
There is something wrong with the dates(years) in the Archives list.
In fact, with 200,000 articles in the DB, the computed dates are huge, in the fake data.
But they are correct. Just try with 200 articles.
do you think is possible the same structure for desktop application?
Yes, it is possible, and even planned.
As I wrote above, still need to write some automated mapping for writing Views as VCL/FMX, and share the same TMVCApplication insteand.
For instance, we may expect UI component names to match the controller (e.g. TEdit/TLabel following parameters names, TButton following methods names).
Then write only some small events in RAD, only if needed.
Offline
@ab
I recall my request.
Can you show how to effectively use external database by ZEOS in this example?
Michal
Offline
Now sample "30 MVC Server" work with an external PostgreSQL server
- either with Zeos/ZDBC or FireDAC libraries.
See http://synopse.info/fossil/info/5569b56688
I discovered that when using Zeos/ZDBC, memory use increases a lot, eating hunderths of MB.
There is no memory leak reported at FastMM4 level in the application, and with FireDAC, it uses small amount of RAM (around 60 MB).
Note that our direct SQLite3 database gives the better performance, use a much lower amount of memory, and consume less disk space for its storage.
Even with 200,000 articles and 400,000 comments in the DB.
External DB is just for demo purpose, here.
Offline
@ab
Thanks.
Michal
Offline
Arnaud,
does this sample use LOB's in any kind?
Offline
@Egon
The "content" text fields (e.g. for blog articles) have no maximum size set - so they are defined as "TEXT".
So I hope they are not handled as CLOB...
If ZDBC handles TEXT fields as LOB, and has problems with it, this is a big issue.
In fact, TEXT is the recommended way of creating a table with some text in PostgreSQL. See http://www.postgresql.org/docs/9.3/stat … acter.html
We should be able to handle "TEXT" columns as fast as possible, even with small content.
Offline
Of course, they are threaded as CLOB, everything else would be wrong. Btw. your could also use VARCHAR with no length.
Hundrets MB of mem... hard to say. NOTE: Zeos internals do cache a loads of of Informations, instead of calling the Server again. Espezially for the slow PostgreSQL-server. This migth be one of Zeos performance advantages against FireDac. But this consumes Memory, of course.
Digging in.. the TZPGResultSet doesn't have a spezial GetUTF8String override yet. Currently i'm working on another refactoring: DO NOT DETERMINE all Columns-Information for !Reopened! reseultsets twice, just reset positions and handles instead. Slightly i've to many locale WC's. But having a look to the miising GetUTF8String override will be next.
May i ask if this example also uses ColumnsToJSON proc?
Last edited by EgonHugeist (2014-10-27 20:10:23)
Offline
No never use CHAR fields IMHO. Better would be VARCHAR(x) fields in all cases. Else you would run either into trailing spaces issues or (this is what Zeos is doing) got a newly performance drop, because we do dec() length of string as long trailing spaces are used!
Instead of, i quickly commited a PGResultSet GetUTF8String override. Patch done R3435 \testing-7.2 (SVN).
This simply suppresse creating LOB buffers and directly return a UTF8String. Hope the Memory usage will decrease a bit?
Offline
@ab
In the version of PostgreSQL(R 1.18.416; ZEOS 7.2.0-beta R 3434(and R 3435)) can not login(Synopse/synopse).
I get an error:
Error Page
Low-level #400 Error occurred with the following message:
Wrong logging information
Error context:
{
"Msg": "Wrong logging information",
"Scope": null,
"main": {
"pageName": "Error",
"blog": {
"Title": "mORMot BLOG",
"Language": "en",
"Description": "Sample Blog Web Application using Synopse mORMot MVC",
"Copyright": "©2014 <a href=http://synopse.info>Synopse Informatique</a>",
"About": "Bla? Ble, bla ble, bli bla ble blu ble. Blo blu. Bla ble, blu bla blu blu bli blu blu bla, bli blo blo blo bli bli blu blo.\r\nBlo bla!"
},
"session": null,
"archives":
[
{
"PublishedMonth": 24368,
"FirstID": 2001
},
{
"PublishedMonth": 24367,
...
Michal
Last edited by miab3 (2014-10-27 22:22:00)
Offline
You typed it correctly?
This error says, that the login information is not correct.
Maybe you've changed something? Another way would be to look up if the User exists in the database.
If there is a User, you just have to know the right password.
Last edited by Pyjama (2014-10-28 06:14:11)
Offline
Thanks Arnaud, will find sometime to look into your MVC design!
Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.
Offline
@ab
Indeed.
But SQLite allowed to log in Synopse / synopse - why?
Michal
Offline
@ab,
A bigger problem is the Firebird by ZEOS.
Not only did to login but does not display Articles.
The database is created, tags are counted.
(I replaced all the LIMIT on ROWS.)
Could you even try the Firebird, please.
Michal
Offline
Instead of, i quickly commited a PGResultSet GetUTF8String override. Patch done R3435 \testing-7.2 (SVN).
This simply suppresse creating LOB buffers and directly return a UTF8String. Hope the Memory usage will decrease a bit?
This sadly did NOT fix the issue!
Now memory still grows, perhaps a little slower, but it quickly reached more than 1 GB.
Whereas FireDAC did use less than 20 MB AFAIR.
Use of PostgreSQL client consumes much more memory than SQLite3 itself, and more that the stored data.
Update:
In fact, I forced to use a single connection for every thread, and now it uses very little memory....
See http://synopse.info/fossil/info/d8240c4f1c
Sounds much more stable now.
Offline
This sadly did NOT fix the issue!
Now memory still grows, perhaps a little slower, but it quickly reached more than 1 GB.
Whereas FireDAC did use less than 20 MB AFAIR.Use of PostgreSQL client consumes much more memory than SQLite3 itself, and more that the stored data.
I know that, but keeps temporary allocated memory much smaller on each GetUTF8String().
Note:
1. since your project works with threads and each thread has a new IZConnection the memory grows because of Cached metadata informations.
2. for PostgreSQL the is a second cache, called TableInfoCache.
supressing the MetaData-Caching:
IZConnection.SetUseMetadata(False);
Well i've commited one little patch R3436 \testing-7.2
suppressing TableInfoCache:
'NoTableInfoCache=True'
to the PostgreSQL-ZURL.Properties. This suppresses caching table informations we added for the NativeResultSets.
All in one.. You wouldn't notice such memory consumtion if there is just one connection...
Offline
Update:
In fact, I forced to use a single connection for every thread, and now it uses very little memory....
See http://synopse.info/fossil/info/d8240c4f1c
Sounds much more stable now.
Offline
That's cool. Looks like you also have a solution for everything.
Note accordingly stable behavior: There is NO difference to see. Just the cached data each connection uses grows the memory.
Anyway, i propose you add my suggested code and some comments to the ZeosDB files, just to notify users about the possibilities...
In addition.. IIRC are you calling the Zeos-MetaData only one on CreateMissingTables, right? So from my point of view the whole caching stuff makes no sence for your framework, Arnaud.
Think about it...
Last edited by EgonHugeist (2014-10-28 14:14:26)
Offline
A bigger problem is the Firebird by ZEOS.
Not only did to login but does not display Articles.
I've added sample MVCServerFirebird.dpr.
Sounds fine on our side, as soon as the DB properties are properly defined.
See http://synopse.info/fossil/info/23ea40f6cd
No need to change the SQL with LIMIT/OFFSET: the ORM will do it for you!
Offline
In addition.. IIRC are you calling the Zeos-MetaData only one on CreateMissingTables, right? So from my point of view the whole caching stuff makes no sence for your framework, Arnaud.
Think about it...
So what we should we do in SynDBZeos?
Is http://synopse.info/fossil/info/ed5b4034df correct?
Offline
So what we should we do in SynDBZeos?
Is http://synopse.info/fossil/info/ed5b4034df correct?
Looks like good start.
To be clear:
Am i right you're calling the Zeos-MetaInformations only once an CreateMissingTables?
If so than you can simply flush the cache by calling:
IZConnection.GetMetadata.ClearCache;
if CreateMissingTables is done. What do you think? You can execute it by default from my point of view..
Offline
Yes, metadata is used only once at startup, in CreateMissingTables.
We may clear the cache.
But is it worth it?
I suspect it has nothing to do with memory consumption during the process, right?
Offline
I suspect it has nothing to do with memory consumption during the process, right?
Hard to say. The two caches MAY consume loads of mem dependend to count of (GetTables + (GetColumns, GetPrimaryKeys/GetIndexInfo) per table)) x CountOf(Connections)
In addition there is the TableInfoCache for PostreSQL only.
So if i would know how this MVC samples do work, i could give you a true answer.. I'm wondering the memory grows during the process..
Propose you test it with multithreading and flushed caches? I would like to know results...
Edit: I really see nothing else which can produce such memory consuming behavior, Arnaud.
Last edited by EgonHugeist (2014-10-28 14:55:35)
Offline
If I clear the cache in sample MVCServerPostgreSQL.dpr:
aServer.CreateMissingTables;
(aExternalDB.MainConnection as TSQLDBZEOSConnection).Database.GetMetadata.ClearCache;
aApplication := TBlogApplication.Create(aServer);
I do not see any difference in memory consumption.
So I would not had it in the main trunk, since it would somewhat difficult to implement, for no benefit.
I guess we still need to do some testing of our solution... in true multi-thread.
Offline
You can safely use both:
IZConnection.GetMetadata.ClearCache;
pesits since i know Zeos.
'NoTableInfoCache=true'
Is a String property which simply has no effect for older Revisions.
During the process only the TableInfoCache can grow the memory until all informations are collected. May i ask if FireBird behaves equal?
Offline
@ab, @EgonHugeist
Thanks for the example of the Firebird. It works and consumes much less memory than PostgreSQL
but now:
mORmot r1.18.428 and ZEOS 7.2 r3436
PostgreSQL also consumes much less memory than previously.
Michal
Last edited by miab3 (2014-10-28 16:10:28)
Offline
@ab
Now, unfortunately, there was an error when (after) Add Comment by synopse:
Error Page
Low-level #404 Error occurred with the following message:
HTTP Error 404 - Not Found
Error context:
{
"Msg": "HTTP Error 404 - Not Found",
"Scope": null,
"main": {
"pageName": "Error",
"blog": {
"Title": "mORMot BLOG",
"Language": "en",
"Description": "Sample Blog Web Application using Synopse mORMot MVC",
"Copyright": "©2014 <a href=http://synopse.info>Synopse Informatique</a>",
"About": "Bla? Ble, bla ble, bli bla ble blu ble. Blo blu. Bla ble, blu bla blu blu bli blu blu bla, bli blo blo blo bli bli blu blo.\r\nBlo bla!"
},
"session": {
"AuthorName": "synopse",
"AuthorID": 1,
"AuthorRights": {
"canComment": true,
"canPost": true,
"canDelete": true,
"canAdministrate": true
},
"id": 2
},
"archives":
[
{
"PublishedMonth": 25168,
"FirstID": 10001
...
Michal
Offline
@ab
r 1.18.433 (SQLite)
Still this error occurs.
After Add Comment, data to write but throws an error as above.
Michal
Offline
Wow I go away for a month and come back to see this... :-)
Great work, showcasing the expandable capabilities of mORMot.
I previous UI generation discussions the focus is always on displaying data, which in my opinion is the easy part.
What I'm interested in is interactive, i.e. INPUT/EDIT of data by users and especially handling concurrency, delta of changes, etc.
E.g. you might have a large TSQLRecord, say "TSQLClient" with many fields and sub-structures (using extensive sharding).
Simple Scenario:
UserA opens Client:ABC001 'view' and go on coffee break, displaying e.g. phone number as '123 456 7890'
UserB opens and edit Client:ABC001 and change phone number to '555 456 7890'
UserB comes back from break, edit 'email address' and save.
Now if I just post everything on UserA 'edit' screen, I will overwrite the phone number change made by UserB.
So in short:
1) how to handle updates and edits from views?
2) how to prevent unexpected data-loss due to concurrent user actions?
Really looking forward to see where this is going.
Best regards
AntonE
Offline
@AntonE
1) This MVC framework is stateless, just like HTTP is.
If you update the page, you will have updated data.
You may add real-time update, via JavaScript - but this won't be part of mORMotMVC.pas yet.
You would rather use a framework like Angular.js on the client side, then let it use the RESTful API of mORMot's server.
But real-time:
- has a performance cost, especially on the server side;
- uses JavaScript, so is much less search-engines friendly than server side generated HTML: mORMotMVC.pas apps are very google/yahoo/bing friendly, whereas an Angular.
About the 2nd point, Google is starting to execute JavaScript when crawling the web - see http://googlewebmastercentral.blogspot. … etter.html
But this is a huge task, and needs a lot of tuning to ensure it will work as expected.
Dynamically created web pages, with several proper levels of cache (as mORMotMVC.pas does), is definitively a more secure solution.
2) This is IMHO to be handled on server side, e.g. at REST level - so when usingRestModel.
At TSQLRecord level, our ORM allows to "lock" a resource (i.e. a TSQLRecord by its ID), when your retrieve it.
Then it will unlock it at update.
See RestModel.Retrieve/Update/Unlock.
For transactions, you have the TSQLRestBatch class which allows atomic commits of several TSQLRecord modifications.
(and may also increase a lot the processing speed, in respect to individual modifications)
Note also that you can automagically store the history of any TSQLRecord, see http://blog.synopse.info/post/2014/06/2 … e-tracking
This would help resolving any conflicts very nicely for the end users.
You can easily display the record history...
Offline
@ab,
R 1.18.438
Stlil Error after Add Comment:
Low-level #404 Error occurred with the following message:
HTTP Error 404 - Not Found
Error context:
{
"Msg": "HTTP Error 404 - Not Found",
"Scope": null,
"main": {
"pageName": "Error",
"blog": {
"Title": "mORMot BLOG",
"Language": "en",
"Description": "Sample Blog Web Application using Synopse mORMot MVC",
"Copyright": "©2014 <a href=http://synopse.info>Synopse Informatique</a>",
"About": "Bla? Ble, bla ble, bli bla ble blu ble. Blo blu. Bla ble, blu bla blu blu bli blu blu bla, bli blo blo blo bli bli blu blo.\r\nBlo bla!"
},
"session": {
"AuthorName": "synopse",
"AuthorID": 1,
"AuthorRights": {
"canComment": true,
"canPost": true,
"canDelete": true,
"canAdministrate": true
},
"id": 1
...
Michal
Offline
@Michal
There was inted small typo in the "30 - MVC Server" blog sample which prevented from adding comments to the page.
Should be fixed by http://synopse.info/fossil/info/4e64baa9ea
Thanks for the feedback!
Offline
@ab
Thanks, now it's fine.
When adding Articles and Users using html
Michal
Offline
@ab
So I was just thinking myself.
For now, we exercise improving performance and reducing memory usage for ZEOS.
Michal
Last edited by miab3 (2014-10-30 20:24:10)
Offline
Note that this sample is tuned for a sqlite db or a MongoDB nosql instance.
External db would be less efficient for instance about the tags dynamic array, which flies on internal sqlite but would be slower with external db, since it uses a blob field.
Offline
@ab
I know that.
But improving ZEOS useful even for other applications.
Besides, sometimes you have to use other than a local SQLite data.
Michal
Offline
Note that this sample is tuned for a sqlite db or a MongoDB nosql instance.
External db would be less efficient for instance about the tags dynamic array, which flies on internal sqlite but would be slower with external db, since it uses a blob field.
I don't want to bother you with this because I know mORMot is muilti-db and you must limit yourself to the lowest common denominator and I am ATM more interested on PostgreSQL support and maybe other people don't care about it, but I have to say it, have you seen the soon to be released 9.4 version and his JSON/JSONB support?
http://opensourceconnections.com/blog/2 … xtensions/
http://blogs.enterprisedb.com/2014/09/2 … r-reality/
I don't know what level of support for this feature will be on Zeos ( hello EgonHugeist ), FireDAC or UNIDAC (I still don't explore this on my own) but it could be worth to take a look at it and perhaps in the future we can have some of these niceties on our little mORMot ?
Offline
We are following PostgreSQL progress on a regular basis.
So we prepared this integration.
The new JSONB format, and its powerful query pattern, would indeed help storing documents, i.e. TSQLRecord published fields stored as JSON, i.e. variant/TDocVariant, record or even dynamic array (once we add http://synopse.info/fossil/tktview?name=27353dad25 feature request).
AFAIR JSONB parameters are just set or retrieved as JSON text, so data access library (e.g. ZDBC) is not to be changed.
Offline
Great! seems like you are always one step forward
Offline
@ab,
All the time I have a problem with the 30 - MVC Server for Oracle and ZEOS (FireDAC).
Do not rename the reserved names and do not INSERT.
Oracle-ZEOS test passes:
http://zeoslib.sourceforge.net/viewtopi … 628#p35628
Michal
Offline
Is it possible to use TMVCApplication with method based services instead of interface based services?
Offline
@esmondb
There is no interface based service here.
In fact there is no service.
The controller is using an interface.
But it is a controller, not a service.
Under the hood, routing is done using something similar to method based services.
Offline