#101 mORMot 1 » TSQLRecordVirtualTableAutoID RowId » 2012-06-14 14:09:11

chapa
Replies: 2

In function:

function TSQLRestServer.URI()
  ...
      if Context.Method<>mLOCK then begin // LOCK not available here
        SQLSelect := 'ID'; // if no select is specified (i.e. ModelRoot/TableName)

We assume that row id column is named 'ID', which is wrong in case when using VirtualTable.
Calling ModelRoot/TableName returns BadRequest, due to wrong SQL query involving column 'ID' instead of 'rowid'.

#102 Re: mORMot 1 » SQLite3 RowID » 2012-06-13 13:15:55

Surely, I agree fully about NativeInt and RowId. It is more common to be able to store negative ids if needed, than unsigned int32 (in case of 32 bit compiler).
Actually, already changed to use Int64. No problems so far, maybe some penalty from RTTI dealing with tkInt64 instead of tkInteger. But think it is a peace of cake in real-life.
Thanks.

#103 mORMot 1 » SQLite3 RowID » 2012-06-13 09:43:22

chapa
Replies: 3

According SQLite3 documentation:

"... In SQLite version 3.0, the rowid is a 64-bit signed integer.
... To minimize storage space, the 64-bit rowid is stored as a variable length integer. Rowids between 0 and 127 use only a single byte. Rowids between 0 and 16383 use just 2 bytes. Up to 2097152 uses three bytes. And so forth. Negative rowids are allowed but they always use nine bytes of storage and so their use is discouraged. When rowids are generated automatically by SQLite, they will always be non-negative."

I noticed that the mORMot framework does not handle well cardinal TSQLRecord published properties, which I use to represent hashes.
If I used to store "real" cardinals directly using sqlite3_xx functions and try to query against TSQLRecord cardinal property which is > MaxInt, than cardinal is represented as negative int and the query does not return proper result.

Of course it can be easily work-arounded, but it will be nice if the framework handle row ids as stated in sqlite3 docs and cardinal properties right.

#104 Re: mORMot 1 » mORMot road-map » 2012-06-11 08:59:37

Hi ab,

Event Sourcing is very interesting topic.
Everyone faced the problems and solutions described.

Reading about Event Sourcing I realize I had implemented it in my own custom way (regardless all the theory).
Will be glad to see your approach and realization.

Fast in-memory database engine is the first thing I may use. Due to the missing in-memory indecies of TSQLRestServerStaticInMemory I dropped it.
Also, implementation can benefit using custom thread data(event) management system, build on SRW Locks. Will break legacy OS.
In any way, the speed and scalability of event backend processing may be the key when taking decision whenever to use such approach.

Will be glad to share my ideas, plans and current sources to mORMot framework, if you find them useful you will need to document and provide some good examples.
They are not so tight around the theory, but may help mORMot users in various use cases.

#105 Re: mORMot 1 » MessageBox » 2012-05-30 09:53:21

Used following Family settings:

    Level := LOG_VERBOSE;
    PerThreadLog := True;
    HighResolutionTimeStamp := True;
    AutoFlushTimeOut := 5;
    OnArchive := EventArchiveSynLZ;
    ArchiveAfterDays := 1;

Will try the new code and report the log to the mail if the problem is still there.
Thanks a lot.

#106 Re: mORMot 1 » MessageBox » 2012-05-30 07:28:52

Wow, last time I checked Smart Studio (less than a month I guess) it was alpha and not available for download.
Arriving just in time, will give it a try.

I am using Delphi X2 Upd1, compiled both logging application and LogView.
Where I can send or post the log file and usage description? Maybe an email?

#107 Re: mORMot 1 » MessageBox » 2012-05-30 06:55:54

Thank you very much for the hints provided.

Currently it is not a big issue for me.
Once I need to communicate over HTTP this will become bottleneck and will be glad to contribute my implementation.

I noticed in LogView application, that some lines are reported with wrong time among others. They had to be chronological.
This leads to wrong time/propertime reports.
If you are not aware of this issue I can provide you with more details and try to look for the problem.

Could someone give me opinion which javascript library prefer/use for rich client of the framework?
jQuery, YUI, or maybe someone other?

Thanks.

#108 mORMot 1 » MessageBox » 2012-05-28 12:53:07

chapa
Replies: 12

Hi ab,

I used to call interface based services over NamedPipe. Using latest code.

Testing the client/server connection failover, where client and server may be both windows services I break into:
SQLite3Commons, line 18317:
MessageBox(0,pointer(SysErrorMessage(Card)),pointer(fPipeName),0);

Isn't it enough just to log the error or pass the exception to the client?
I think, that no UI modal dialogs have to be shown by the framework, especially at low-level.

Another question:
It is very handy to specify custom service answer, allowing to reply with compressed/binary content and bypass MIME encoding.
Is there an easy, build in the framework, way for the client to send binary content to the server service?

Thanks.

#109 Re: mORMot 1 » sqlite3_step memory leak in case of SQLITE_CONSTRAINT » 2012-05-09 14:04:11

Hi ab,

It is very easy reproducable:

sqlite3_prepare_v2(fClient.DB.DB, testsql, -1, fTestStmt, fTestTail)
sqlite3_reset(fTestStmt); //With or without, same leaks
sqlite3_finalize(fTestStmt);

StackTrace is:
4046FA [System][@GetMem]
558200 [SynSQLite3.pas][SynSQLite3][@Synsqlite3_malloc][2446]
55D783 [SynSQLite3]
55DF07 [SynSQLite3][@Synsqlite3_sqlite3_memory_highwater]
564167 [SynSQLite3][@Synsqlite3_sqlite3_realloc]
...

Can you reproduce it?


PS: Excuse me. All the problems seems that the Client (fClient.DB.DB) is disposed before the sqlite3_finalize.
Fixed.
Very sorry for bothering you.

#110 Re: mORMot 1 » sqlite3_step memory leak in case of SQLITE_CONSTRAINT » 2012-05-08 07:53:55

Got it.

define:
fKeySelectRequest: TSQLRequest;

on create:
fKeySelectRequest.Prepare(fClient.DB.DB, 'SELECT ID FROM TABLE WHERE COL=@1');

on destroy:
fKeySelectRequest.Close

is leaking if no step iteration called.

#111 mORMot 1 » sqlite3_step memory leak in case of SQLITE_CONSTRAINT » 2012-05-08 07:36:33

chapa
Replies: 3

Hi,

When using TSQLRequest.Step or directly sqlite3_step() and in case of SQLITE_CONSTRAINT error result over prepared statement FastMM detects memory leaks on shutdown.
The leaks are always constant in size, does not depend on sqlite3 step iterations.

Table: CREATE TABLE Key(ID INTEGER PRIMARY KEY AUTOINCREMENT, CreateTime INTEGER, ModTime INTEGER, Hash INTEGER, Name TEXT COLLATE SYSTEMNOCASE)
Index: CREATE UNIQUE INDEX IndexKeyHash ON Key(Hash)
Prepared statement: 'INSERT INTO Key VALUES (NULL, @1, @2, @3, @4)'

The leak happens if I try to violate IndexKeyHash

sqlite3_finalize(stmt) is called, and it seems I am doing nothing wrong, as far as I can see.

Is someone aware of this problem, or can reproduce it?
Thanks.

#112 Re: mORMot 1 » TestSQL3 fail » 2012-05-03 07:52:36

Did not played much with nod32 options.
There are exclusions list for specific domains, which can be set for not scanning. Did not tried adding localhost or remote one in exclusion list.
Also disabling in-depth heuristics and advanced heuristics may lead to much better performance, as nod32 will not try to interpret custom compressed content as binary executable and try to detect low-level malware code inside the "unknown" binary content type.
By default HTTP checking is enabled only for "80, 8080, 3128". I used two >1024 different ports for tests, but there is no effect till whole HTTP checking is disabled.

My use case is not common.
On middle-tier it is more like data processing than ORM, but think I can benefit from using ORM framework, even if I may had little performance impact compared to current custom realization.
On server side I had dozen of tables, few of them with a couple of millions objects. Nothing special. Here I will benefit from using mORMot, with fast SQLlite3 engine and caching. RESTful services are what I need there.
But, middle tier need something fast, in-memory indexed realization, which receive from programs (clients) around 300-500k of new generated objects (only content, no internal ids) every 5 minutes.

Must check:
Does already have them in-memory with assigned server id?
  yes (99.9%): check if content changed
    yes, changed (<1%): update the object to the Server tier, using known id and new changed content
    not changed (>99%): do nothing
  no (<0.01%): add them to the server, store in-memory the content along with new server id

As far as new and changed objects are very little % of all processed incoming objects I think it is good to be separate tire.
If cache is good enough it will be easier to use client-side cache of server objects, but I guess the purpose of the cache is other, and realization is not fast enough to process such number of objects efficiently.
I would like to implement it having mORMot functionality, maybe bigtable is suitable for this midware? Or other mORMot implementation?

#113 Re: mORMot 1 » TestSQL3 fail » 2012-05-02 14:34:03

Actually, I found what the problem is.

Nod32 is catching and processing every HTTP like request, even if the port is not in the list of "Ports used by HTTP protocol". Framework compression is very "suspicious" and processing of http compressed content seem very slow compared to regular HTTP traffic.

Disabling Nod32 HTTP filters fixed the mess. Good to know.

One question. Is TSQLRestServerFullMemory aware of CreateSQLMultiIndex? Query using TSQLRestServerFullMemory is very slow compared to TSQLRestServerDB with created indecies.

Thanks.

#114 Re: mORMot 1 » TestSQL3 fail » 2012-05-01 18:52:44

Yes, sometimes I use proxy, this is the first thing I checked then I saw the problem. But proxy is not configured right now.
May it be the case? Maybe if there is a way to reset all WinHTTP settings will be good.

Will try to find out the exact problem, because I want to get familiar with the framework and give it a try even in production.
Also will be more than happy if I got a chance to contribute.

Thanks, will try also WinINet and WinSocket with the same tests and let you know the results.

#115 Re: mORMot 1 » TestSQL3 fail » 2012-05-01 14:13:41

Thanks, easy fix smile Guess pdf fails are related to WinAnsiToUTF8, will not trace the test.

I am experiencing another major issue. Before half an year I tried some of the Samples. Found major performance problems and I give up right after first tries.

Now, after reading the SAD I got excited. Good work done.
But I faced same overall performance problems and just traced the problem.

SynCrtSock.pas
3989:
function WinHttpSendRequest(hRequest: HINTERNET; pwszHeaders: PWideChar;
  dwHeadersLength: DWORD; lpOptional: Pointer; dwOptionalLength: DWORD; dwTotalLength: DWORD;

Dont know why, but on my machine calling this function is EXTREMELY slow. Sometimes it takes more than a second for just 2-3 calls?!?
All antivirus programs stopped, if may be related.

Tried on other Win7 and Win2008RC2, and I does not face same issue. All worked fine.

Did anyone face such problems before? What could be the reason of very low winhttp.dll performance?

Thanks.

#116 Re: mORMot 1 » TestSQL3 fail » 2012-05-01 12:55:34

I am running Win7 64bit, Delphi XE2 Update 1.
Already running latest version from fossil. No other(old) versions around.

Last 10th test failed, in TTestLowLevelCommon._IdemPropName, found in SynCommons.pas:
22546: Check(UpperCaseU(WinAnsiToUTF8('aйзD'))='AECD');

My default locale is Cyrilic (Win-1521).
Tried to convert the SynCommons.pas directly from IDE to UTF8 (File Format), also to add UTF8 BOM to get the IDE read it as UTF8 text. Same failed results.

Could this be the problem?
If yes, how can I get around this?

Other tests are also failing (pdf; one client-server test lasts forever). Will post them once I resolve this issue. I guess all the fails may be related to the problem above.

Thanks.

#117 mORMot 1 » TestSQL3 fail » 2012-04-30 14:26:28

chapa
Replies: 11

Hello,

I Need assistance.

Would like to try the framework, but unfortunately running TestSQL3 in last few leaves I got:

O:\Components\Synopse\SQLite3\TestSQL3.exe 0.0.0.0 (2012-04-30 17:15:54)
Host=HOSTX User=USERX CPU=2*9-6-5894 OS=13.1=6.1.7601 Wow64=1 Freq=2467851
TSynLog 1.16 2012-04-30T17:19:45

20120430 17194558 fail  TTestLowLevelCommon(0216A8E8) Low level common: IdemPropName "" stack trace 0007431B SynCommons.TTestLowLevelCommon._IdemPropName (22547) 001839D2 SQLite3SelfTests.SQLite3ConsoleTests (217) 000076C4 System.@StartExe

What could be the problem?

Thanks.

Board footer

Powered by FluxBB