#1 Re: mORMot 1 » Dangerous IgnoreSSLCertificateErrors defaults » 2021-03-02 14:51:58

Hi,
I would like also to be able to retrieve server certificate info. Is there a way to do so?
Having this, I can check whenever there is a proxifier and/or fiddler between the App and the Server to sniff https traffic.

#2 Re: mORMot 1 » THttpApiWebSocketServer Features Requests » 2019-03-02 09:48:41

@mvp
Looking at http.sys websocket implementation, I feel it very close and native to me, as I remember well back on 2013, when I fight making it work. I would like to give it a try with mormot, and already had adopted it, now serving real-time communication between internal services stable, at extremely high rate and no latency. So, thank you very much for your and Vadim contribution, now it is not sitting in the corner smile

Speaking about it, I was always wondering about migrating on Linux. Recently was reading a lot about RabbitMQ also.
But I did not felt comfortable going this way. I am stuck with Window's SRWLock, ConditionVariables, Queues. Will have hard time moving all of them, and to be honest, I am very satisfied about their performance.

I am very keen to ask you about your impression, pros and cons migrating services to linux, and rabbitmq performance itself in terms of throughput and latency? I am sure you put it on a stress smile

#3 mORMot 1 » THttpApiWebSocketServer Features Requests » 2019-02-28 06:43:45

chapa
Replies: 8

Hello ab,

I would like to share some thoughts about WebSocketServer, as I am about to run it on production server, and face some problems which I am workarounding but is better to be handled by the framework itself.

1. New callbacks for OnPing and OnPong frames. Let us define our App logic around this mechanism. Very handy when measuring connection speed, as sending messages over slow connections easy fills http.sys kernel side entity cache. When it happen it closes underlying TCP connection, which leads to problems to the web socket client and THttpApiWebSocketServer itself. Pinging client and waiting for pong response every XXX kb transferred until we continue with transfers is an easy fix.
2. Better handle Http.ReceiveRequestEntityBody and Http.SendResponseEntityBody for errors, eg.  ERROR_NETNAME_DELETED for read, which often precedes ERROR_CONNECTION_INVALID for write, when underlying tcp connection broke.
3. Better synchronization of TSynWebSocketGuard actions. When I used to run it, I often end with application process being terminated with "Faulting module name: KERNELBASE.dll". Further investigation shows:
STACK_TEXT: 
KERNELBASE!RaiseException+0x68
websocket!ReportFatalException+0x74
websocket!WebSocket_invalid_object_handle_fatal_error+0x32
websocket!WebSocket::VerifyCookie+0x1c
websocket!WebSocketSend+0x24
Debugging shows it happens when guard close the connection and handle to the websocket is set to nil, but TSynThreadPoolHttpApiWebSocketServer.Task still try to do number of WebSocketAPI operations over a nil handle
4. If possible some mechanism to wait until http drain its response entity body, like socket flush

Will stop here smile

Thanks.

#4 mORMot 1 » DateToIso8601PChar when year is not 4 digit » 2019-02-11 08:40:08

chapa
Replies: 1

Hi ab,

Recently debugging why I am producing invalid JSON content led me to following:

using ObjectToJson and more precisely
procedure DateToIso8601PChar(P: PUTF8Char; Expanded: boolean; Y,M,D: cardinal);
expects the year to be 4 digits, reasonably right expectation.

But in practice, I am transferring objects, containing TDateTime, which are filled/generated based on public data, and there is no guarantee that the year will be correct parsed/provided.
I got some 50895 year, if I remember well, than #0 in the json content.

As I believe we will use mORMot even after 9999 year, so I gladly ask for a fix big_smile

P.S.
I still generate invalid UTF8 when streaming data, will further investigate.

#5 Re: mORMot 1 » Potential AV in _JsonFast » 2019-01-21 09:14:05

@ab I can confirm I get no more random AVs as described!

Thanks once again for the patch.
And to @mvp for finding it, I was unable to and got used to live with it smile

My build is XE3 CE x64 on various Windows.

#6 Re: mORMot 1 » Potential AV in _JsonFast » 2019-01-09 15:29:00

yeah, past three years I experience random AVs as described. One on million, but still random AVs, was unable to track them.

mvp, does it resolves this old timer in your opinion:
https://synopse.info/forum/viewtopic.php?id=3283

Just compiled the new sources, and cross fingers for next 24 hours smile

#8 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-03 10:43:32

I guess the code

      while item[itemcount].FromNext(row.Item.Data.DocList) do begin
        inc(itemcount);
        if itemcount>itemsize then begin
          inc(itemsize);
          Setlength(item,itemsize); // a field was deleted from TSQLRecord
        end;
      end;

in some cases item[itemcount] can be out of range, addressing "future" record while row.Item.Data.DocList is not at its betEOF, leading in assigning TBSONElement record properties in not assigned heap. Later increasing item array Length may lead to unpredictable memory behaviour on next iterations on same heap area.
Or something like this smile

#9 mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-03 10:13:41

chapa
Replies: 10

Hi ab,

Running latest mORMot, XE7 64bit build and FastMM4991 FullDebugMode enabled.

Getting:

FastMM has detected an error during a FreeMem operation. The block footer has been corrupted. 

The block size is: 376

This block was allocated by thread 0x38C8, and the stack trace (return addresses) at the time was:
425A28 [FastMM4.pas][FastMM4][_ZN7Fastmm411DebugGetMemEx][8737]
4092AD [System][_ZN6System11_ReallocMemERPvx]
415529 [System][_ZN6System17DynArraySetLengthERPvS0_xPx]
415741 [System][_ZN6System18_DynArraySetLengthERPvS0_x]
E70EA6 [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB13GetJSONValuesEN6System11AnsiStringTILt65535EEENS1_12DynamicArrayINS2_ILt65001EEEEEPN6Mormot15TJSONSerializerE][988]
E745AD [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB10EngineListEN6System11AnsiStringTILt65001EEEbPx][1302]
9AA8E5 [mORMot.pas][mORMot][_ZN6Mormot14TSQLRestServer10EngineListEN6System11AnsiStringTILt65001EEEbPx][40319]
9874E4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest11ExecuteListEPKNS_15TSQLRecordClassEiN6System11AnsiStringTILt65001EEE][33845]
983153 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest16MultiFieldValuesENS_15TSQLRecordClassEN6System11AnsiStringTILt65001EEES4_][32944]
98385E [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPNS_10TSQLRecordES3_][33004]
9848F4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPKNS1_7TVarRecEiS6_iPNS_10TSQLRecordES3_][33217]

The block is currently used for an object of class: Unknown

The allocation number is: 439210

The current thread ID is 0x38C8, and the stack trace (return addresses) leading to this error is:
425C08 [FastMM4.pas][FastMM4][_ZN7Fastmm412DebugFreeMemEPv][8857]
409235 [System][_ZN6System8_FreeMemEPv]
41595D [System][_ZN6System14_DynArrayClearERPvS0_]
E7172D [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB13GetJSONValuesEN6System11AnsiStringTILt65535EEENS1_12DynamicArrayINS2_ILt65001EEEEEPN6Mormot15TJSONSerializerE][1037]
E745AD [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB10EngineListEN6System11AnsiStringTILt65001EEEbPx][1302]
9AA8E5 [mORMot.pas][mORMot][_ZN6Mormot14TSQLRestServer10EngineListEN6System11AnsiStringTILt65001EEEbPx][40319]
9874E4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest11ExecuteListEPKNS_15TSQLRecordClassEiN6System11AnsiStringTILt65001EEE][33845]
983153 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest16MultiFieldValuesENS_15TSQLRecordClassEN6System11AnsiStringTILt65001EEES4_][32944]
98385E [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPNS_10TSQLRecordES3_][33004]
9848F4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPKNS1_7TVarRecEiS6_iPNS_10TSQLRecordES3_][33217]

It seems

item: array of TBSONElement 

got corrupted, when

item[itemcount].FromNext(row.Item.Data.DocList) 

is iterated at some point

manually setting SetLength(item, 0) at the end or waiting "end;" compiler generated code is leading to memory corruption according FastMM4

#10 Re: mORMot 1 » Record Load/Save JSON Thread Safe? » 2016-04-21 14:32:25

Thanks ab,

Along with not obvious AVs, I got this madExcept log:

exception message  : Access violation at address 0000000000411D75 in module 'test.exe'. Write of address 0000000000FACCF0.

thread $1184:
00411d75 +015 test.exe System                        @LStrClr
0080fff6 +0a6 test.exe SynCommons         34391   +5 TJSONCustomParserRTTI.FinalizeNestedRecord
008100e4 +044 test.exe SynCommons         34421   +7 TJSONCustomParserRTTI.FinalizeNestedArray
00810126 +016 test.exe SynCommons         34429   +1 TJSONCustomParserRTTI.AllocateNestedArray
008105a3 +303 test.exe SynCommons         34493  +29 ProcessValue
00811320 +510 test.exe SynCommons         34668  +64 TJSONCustomParserRTTI.ReadOneLevel
00811e47 +037 test.exe SynCommons         34818   +2 TJSONRecordAbstract.CustomReader
0080dee3 +2b3 test.exe SynCommons         33861  +19 RecordLoadJSON

Please, note, that I do not have any string properties defined in the record!

I think RecordLoadJSON in some odd cases brokes input buffer and/or its own props buffer.
Even in other records containing RawUTF8 I got broken reference to them causing system finalizearray to AV on record cleanup upon procedure end.

Record definition is as follow:

  TTestDTO = packed record
    Id: Integer;
    Seen: Integer;
  end;

  TTestArrDTO = packed record
    arr: array of TTestDTO;
  end;

  __TTestDTOJSON =
    'Id Integer ' +
    'Seen Integer';

  __TTestArrDTOJSON =
    'arr [ ' + __TParserOddSeenDTOJSON + ' ]';

  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestArrDTO), __TTestArrDTOJSON).Options := [soReadIgnoreUnknownFields];

Using Delphi XE7, Debug 64bit

When I thread protect the call to RecordLoadJSON() everything goes just fine.

Any input where and how I should proceed will be appreciated.

#11 Re: mORMot 1 » Record Load/Save JSON Thread Safe? » 2016-04-20 19:40:58

Thanks ab,

Than, going to investigate further the AVs at SynCommons unit and let you know results.

About the performance, are Record Load/Save JSON functions fastest approach when it comes to object<->json conversion?
How slower in your opinion using _Json and TDocVariants are compared to them?

Thanks.

#12 mORMot 1 » Record Load/Save JSON Thread Safe? » 2016-04-20 19:07:12

chapa
Replies: 4

Hi,

Are RecordLoad/SaveJSON() thread safe?

I have really high multithread load using them. Always had random AVs, but recently they became more and more often.
Did secure them inside a lock, and everything seems fine now.

I had random AVs at TTextWriter.AddRecordJSON, TJSONRecordAbstract.CustomWriter, TJSONCustomParserRTTI.WriteOneLevel, WriteOneValue.
Also at FinalizeRecord and LStrClr when local variable of the record is about to be released ending a procedure call.

Also wanted to ask, are this methods fastest approach when it comes to object<->json conversion?

Thanks.

Regards,
chapa

#13 Re: mORMot 1 » WinHTTP, http.sys and WebSockets » 2016-01-29 22:42:44

Great News!

Can not wait to play with the sources.

Long time ago I made some tricks using IOCP and dwsHTTPSysAPI (at that time it supported http.sys 2 over mORMot) and Win8 WebScoket API
Don't know if it will be in any help now but working code was posted 2 years ago at: https://code.google.com/archive/p/dwsockets/

Hope that may help

#14 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-05 11:32:00

Indeed, TEXT is preffered and recommended.
There are many other "on the fly" methods in postgresql to limit the length later if needed at almost no penalty.

Adding dPostgreSQL in DB_HANDLEINDEXONBLOBS did the trick, now multi indexes are created right.
Next problem is that CreateSQLMultiIndex throws an exception on second program run.
As far as I know, there is no statement IF NOT EXISTS for index creation in Postgresql, hope I am wrong.

But ZEOS should have information about current table indexes, and by its name mormot can determine whenever to try create the index or not?
Or maybe do as described here http://dba.stackexchange.com/questions/ … -not-exist for Postgresql 9.0+

#15 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-05 10:22:51

Ok, here come example:

program testmultiindex;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  SynCommons,
  SynDB,
  SynDBZEOS,
  SynSQLite3,
  SynSQLite3Static,
  mORMot,
  mORMotDB,
  mORMotSQLite3,
  System.SysUtils;

type
  TSQLTestRec = class (TSQLRecord)
  private
    fTestInt: Integer;
    fTestRawUTF: RawUTF8;
  public
  published
    property TestInt: Integer index 50 read fTestInt write fTestInt;
    property TestRawUTF: RawUTF8 index 50 read fTestRawUTF write fTestRawUTF;
  end;

var
  props: TSQLDBConnectionPropertiesThreadSafe;
  model: TSQLModel;
  server: TSQLRestServerDB;

begin
  {$WARN SYMBOL_PLATFORM OFF}
  ReportMemoryLeaksOnShutdown := True;
  {$WARN SYMBOL_PLATFORM ON}

  props := TSQLDBZEOSConnectionProperties.Create(
    TSQLDBZEOSConnectionProperties.URI(dPostgreSQL,'remoteip:port'), 'dbname','user,'pass');
  try
    props.ThreadingMode := tmMainConnection;
    props.ForcedSchemaName := 'raw';

    model := TSQLModel.Create([TSQLTestRec], 'raw');
    try
      VirtualTableExternalRegister(model, TSQLTestRec, props, '');
      model.SetCustomCollationForAllRawUTF8('BINARY');

      server := TSQLRestServerDB.Create(model, SQLITE_MEMORY_DATABASE_NAME);
      try
        server.CreateMissingTables;
        server.CreateSQLMultiIndex(TSQLTestRec, ['TestInt', 'TestRawUTF'], True);
      finally
        server.Free;
      end;
    finally
      model.Free;
    end;
  finally
    props.Free;
  end;

  ReadLn;
end.

With or without "index 50" attribute same table structure generated which can be described as:

CREATE TABLE raw.testrec (
  id INTEGER NOT NULL,
  testint BIGINT,
  testrawutf TEXT,
  CONSTRAINT testrec_pkey PRIMARY KEY(id)
) 
WITH (oids = false);

CREATE UNIQUE INDEX ndxtestrectestint ON raw.testrec USING btree (testint);

There is no difference if "index 50" property attribute is set or not, seem same "testrawutf TEXT" definition used, instead of "testrawutf VARCHAR(50)" in index case.
All the time same single column index created, instead of two column multi index over  ['TestInt', 'TestRawUTF']

Hope this will convenience you to look into this problem.

#16 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-04 19:47:22

thanks, and please excuse me.
will try tomorrow limiting text size with limit attribute, making it varchar I guess, in order to be involved in multi index.
than will try  create multi field index again.

#17 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-04 14:56:38

indexing on columns defined as "a:integer; b:rawutf8; c: rawutf8" is indexing by blob fields?
if yes, would like to report it.

#18 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-04 14:10:06

ok, to be more precise:

here is the case when length(FieldNames) <> 1 in TSQLRestStorageExternal.CreateSQLMultiIndex():

  if not (fProperties.DBMS in DB_HANDLEINDEXONBLOBS) then
    // BLOB fields cannot be indexed (only in SQLite3)
    for i := 0 to n-1 do begin
      extfield := fFieldsInternalToExternal[IntFieldIndex[i]+1];
      if (extfield>=0) and
         (fFieldsExternal[extfield].ColumnType in [ftBlob,ftUTF8]) and
         (fFieldsExternal[extfield].ColumnLength<=0) then begin
        if i=0 then
          exit; // impossible to create an index with no field!
        //SetLength(ExtFieldNames,i); // truncate index to the last indexable field   // commend by chapa
        break;
      end;
    end;

Imagine we index on "integer,text,text"
In cycle we pass i=0 as condition fFieldsExternal[extfield].ColumnType in [ftBlob,ftUTF8] I think should not met.
Than, on next iteration, when i=1 I guess above if condition is met, we get into begin ... end, but i<>0 and we do not exit, but break the cycle setting length of the ExtFieldNames to 1 (leaving only first column to be indexed).
Later on "SQL := fProperties.SQLAddIndex(fTableName,ExtFieldNames,Unique,Descending,IndexName);" here ExtFieldNames point only on index "integer", but not "integer,text,text".

#19 Re: mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-04 13:34:23

I do not think it is related to ZEOS, but to mORMotDB.pas:

1582: function TSQLRestStorageExternal.CreateSQLMultiIndex()
1619:     for i := 0 to n-1 do begin
1626:         SetLength(ExtFieldNames,i); // <- here I suppose all fields right are trimmed

In other case I would provide example, but here I think there is no need.

#20 mORMot 1 » ZEOS+Postgresql CreateMultiIndex » 2014-11-04 12:24:54

chapa
Replies: 12

Hi ab,

CreateMultiIndex() in case of ZEOS external Postgresql does not make multi index, but only with first column.
Also, it does not check if index name already exists (including the case when index name looks like some hash).

#21 Re: mORMot 1 » Async Insert/Updates » 2014-09-16 14:25:19

hi ab,

You understand me well, only I should expand it a bit at "Only send the modifications once in a while to the DB, e.g. once the record set content has been "stabilized"" it can be an option when to sync when update the record.

Update the record and:
1. Immediate sync
2. Schedule sync after X milliseconds (on every update it will reschedule it, but sync if timeout)  (this may break current design principles, if background thread manage it)
3. Background thread doing sync regarding above 2 settings if DB backend is idle and/or memory limit reached. On batch portions. (this definitely breaks current design principles)

Use cases may be:
1. any Forex trading systems, which will keep track of any rate change, keep change history, but without hitting the DB for every change, meantime all clients will have up to date rates which is critical for such systems.
2. Real-time network dump statistic system is also candidate for it. (ex. You may miss exact total bytes sent/received, but you do not have to hit the DB for every count network packet, or to log every packet size in order to later sum it).
3.Any other real-time processing and reporting system.
Such system may process tens of thousands updates per second for a set of records.

My use case is different, but I guess much more complicated. I do scan/parse thousand web sites, extract "rates" and process them. This is done in 1-3 seconds interval for each site and extracted entities (records) may easy hit 50000/second.
Persisted change history is essential (change miss not critical thought), along with client to have real-time up to date information.

I have custom solution over mORMot, and it will be really really good if mORMot have such general purpose functionality on the fly.

#22 Re: mORMot 1 » Async Insert/Updates » 2014-09-16 13:19:26

Indeed, I was not clear enough.
I am aware of caching mechanism of mORMot and multithreading.

Take into consideration following use case.

We have one writer and many readers. Writer try to pass real-time data source, which changes really often (50000 times per second), and latest (most up to date) state need to be able to be read from clients with as less latency.
Only recent state should be kept in the DB.

Writer can insert/update/delete at high rate. All record states can be cached for later sync with DB.
In this case 1000 updates/sec on one record can later touch DB for update only 1-2 times, with most recent state for sync.
Meantime, upon client request, server will return either from DB/internalcache, or most recent record for sync, which is not already persisted with DB.

The purpose and final goal is, with compliments to DB's "DiskSync=Off" or common DB fault tolerant mechanisms, we will have super fast in-memory sync&flush structures and possibility to update/insert and query records at really fast rate, to reflect real-time data, and only most recent record (or even only modified properties of records (column)) to be synced (flushed to db) at later time (after period of time, after number of changes, etc).

Of course this may be set up only for one table, where we can afford data loss of last record states for sync in case of disaster. They may even flush to binary file upon shutdown and later come up into the memory if server restart occur (like mORMot's Memory Tables)

All above can of course be done with custom in-memory data structures without overhead of mORMot, but it will be custom solution. Much better to have REST and mORMot over it.
Such a functionality will be Redis killer for Delphi programmers.

I know there is TSQLRestStorageInMemory and similar, which are fast, but are still not a real DB.
As one may have millions of records which are synced into a DB, and at a time only 100 records are "live" and updated at high rate.
Currently, can not load millions records into TSQLRestStorageInMemory nor update DB backed storage at high rate.
Maybe TSQLRestStorageInMemory which syncs with corresponding RestStorage and some manager in front of them will do the trick.

Such implementation may later be used for replication.

Hope it is more clear now smile

#23 mORMot 1 » Async Insert/Updates » 2014-09-16 10:54:50

chapa
Replies: 8

Hi ab,
Is there any build in functionality or plan to do async/postponed/cached CRUD operations against mORMot server?

#24 Re: mORMot 1 » ZEOS + Postgres Remote Server » 2014-09-15 13:32:25

Also,

CreateMissingTables() seems to explicit create unique index by id:
CREATE UNIQUE INDEX ndxsamplerecordid ON public.samplerecord USING btree (id);

but Postgresql creates index on the fly when PrimaryKey Constraint specified in Create Table:
CREATE TABLE public.samplerecord (
  id INTEGER NOT NULL,
  ...
  CONSTRAINT samplerecord_pkey PRIMARY KEY(id)
)

All tables end up with duplicate index for primary id key.
Extra index, extra disk size, extra row update time.

#25 mORMot 1 » ZEOS + Postgres Remote Server » 2014-09-15 12:53:12

chapa
Replies: 3

Hi ab,

Just start playing with ZEOS + Postgresql.
What I find out is when try to set remote server and/or port in TSQLDBZEOSConnectionProperties.URI() class function where if aLibraryLocation='' then we exit and miss the chance to set result+'//'+aServerName

#27 Re: mORMot 1 » hello world mustache » 2014-05-09 07:43:58

Here I should agree with Eric, HTML rendering in modern web applications should be done on client side, and hard data manipulation and serving at server side.
At least this is what most applications do, retrieve the data (binary or json) from a server (most likely not sql but rest), and visualize it depending on its kind (native win, native mac, browser web site, browser app).

We have great and powerful framework at server side, no doubt.
But following tendencies and client's requests, it is more common to build client applications not using delphi native apps, nor C#, but thin js+html+css apps, enabled on all devices, backed up by powerful data server.

What will be in great benefit for the community, in my opinion, is a javascript client framework and/or library, easy interacting with mORMot server.
There a build-in templating will be at great benefit, but it is little peace of what such a framework should be.

#28 Re: mORMot 1 » AV at mORMot.pas » 2014-05-09 06:41:02

Multithreaded x64 debugging is so hard ... ide crashes at random

Anyway, trying to ObjectToJSON() an object, which contains TObjectList as a property.
The AV is on the next line 31130 at TJSONSerializer.WriteObject:

if (not TSQLRecord(Value).fFill.JoinedFields) and (P^.PropType^^.ClassSQLFieldType=sftID) then

here Value if of class TObjectList, recognized as Kind tkClass.
but why you try to cast this object as TSQLRecord and try to access a field of it (fFill in this case)? this definitely will lead to random AV ...

Simple fix works if compiler optimizations is turned on and exchange the if's AND statements like:

if (P^.PropType^^.ClassSQLFieldType=sftID) and (not TSQLRecord(Value).fFill.JoinedFields) then

At run-time first will be checked P^.PropType^^.ClassSQLFieldType which is sftUnknown and skip erroneous cast and go the else clause and finally return json without the AV

Hope my explanations are understandable and will help.

#29 Re: mORMot 1 » AV at mORMot.pas » 2014-05-08 13:54:09

Had to go out of the office now, tomorrow will isolate simple code, test data and figure out the problem, if still needed.

#30 mORMot 1 » AV at mORMot.pas » 2014-05-08 13:47:09

chapa
Replies: 4

Just updated the sources (was 2 months old).

Trying TServiceFactory.Get().Add(TObjectList) got:
Access violation at address 00000000009E139F in module '.exe'. Read of address FFFFFFFFFFFFFFFF

at mORMot.pas line 31130:
tkClass: begin
  Obj := pointer(GetOrdProp(Value,pointer(P)));  // works also for CPU64 <- AV here

It happens only on x64 build, 32bit works fine, and is due to something changed last few months (worked fine before source update).
Can debug further if needed.

#31 Re: mORMot 1 » DynArrayLoadJSON » 2014-02-21 06:59:44

I confirm the fix works in my case smile thanks

#32 Re: mORMot 1 » DynArrayLoadJSON » 2014-02-20 19:37:01

Thanks ab,
Yes, source json looks like the one you typed, with many spaces and #10 #13.
Will test it tomorrow over my source json and let you know results.
Thanks once again.

#33 Re: mORMot 1 » DynArrayLoadJSON » 2014-02-20 14:57:32

Forgot to mention, it happens when soReadIgnoreUnknownFields is set and not all property names are defined in the descriptor constant string.
Maybe this will lead you to some ideas.
Will email example tomorrow.
Thanks.

#34 mORMot 1 » DynArrayLoadJSON » 2014-02-20 13:34:35

chapa
Replies: 5

Hi ab,

Very happy with new record <-> json functionality.
Trying to DynArrayLoadJSON valid JSON string I get wrong/no result.

Figured out, that if I delete all #13 #10 #9 chars from source json string everything goes correct.
It is in case where I have array of record inside array of record.

#35 Re: mORMot 1 » OFF-TOPIC: Merry christmas... » 2013-12-25 09:56:43

Честита Коледа to all! Wish you all the best.

#36 Re: mORMot 1 » Obscenities with mORMot and SmartMS » 2013-12-08 20:52:06

It was fun explaining my wife what mORMot is all about watching the demo at Sunday family's dinner time.
Now, my 4 and 5 years girls have more clear idea about the work at the office during daytime smile ...

#37 Re: mORMot 1 » Server side validation returning all errors » 2013-11-26 12:59:31

+1 for such functionality. Server side validating should be part of the framework.

#38 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-30 17:05:42

Forgot to mention, fiddler also tried, no Content-Encoding header in the response, body is decompressed.
Fiddler is a proxy, they use .NET classes to access Inet, AFAIK.

Once I used to write protocol handlers. Before response hit the client it can be manipulated by several handlers. So I guess it is something like that sitting between the client and actual response.
For protocol handler point of view it does not mind who the client is.

Will give a try to wireshark tomorrow and let you know.

#39 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-30 16:47:48

1. HTTP.SYS one.
2. application/json
3. debugged and ensured it returns compressed content and size

Chrome, Firefox even command line wget.exe returns uncompressed data (wget with Accept-Encode headers set). Tried connecting from localhost to: 1.localhost (for debugging) and 2. remote server (production).
But ... when accessing my computer and production server from other machines, data is compressed smile ...

This leads my mind, that some antivirus or other http protocol handler on localhost catches the response and manipulate it.

#40 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-30 13:00:29

I believe it is some protocol handler on my win box.
Tried to disable the antivirus, but no luck, same problem.

Some protocol handler seem to decompress on the fly the http compressed response content and serve it uncompressed skipping Content-Encoding: deflate header from the http response.
If someone know how I can list/trace all protocol handlers in order to catch who is doing this will help a lot.
Thanks.

#41 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-29 16:50:46

thank.
using latest firefox and chrome.
right headers sent, but response is not compressed.

#42 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-29 14:51:53

ab, after updating the source with the new TSQLRestServerURIContext I got following:

20131029 16374619  " EXCOS EAccessViolation (C0000005) at 001CF809 mORMot.TSQLRestRoutingREST.URIDecodeSOAByInterface (24348)  stack trace API 001865C4 SynCommons.SynRtlUnwind (31512) 0000945C System.@HandleOnException
i := Server.Services.fListInterfaceMethod.IndexOf(URI); -> here Server.Services are nil

This happen every time when trying to call invalid (non-existing) method name of the rest server (and no interface services defined).
Maybe some handled error will be more suitable than showing AV exception with debug information to the client.

Also I was unable to get compression working.
Defined in the project COMPRESSDEFLATE, also tried to manually RegisterCompress, but with no luck.
Browser request compression, but the server never compress (tried also with big json as response).

#43 Re: mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-29 13:58:30

right, it works with latest sources (after BREAKING CHANGE)
thanks.

#44 mORMot 1 » ModelRoot/TableName/ID/MethodName » 2013-10-29 12:55:46

chapa
Replies: 13

Hi ab,

It is very handy to call Method based services and have TSQLRecordClass and record id in the TSQLRestServerCallBackParams context.

Is it possible to implement something like "ModelRoot/TableName/MethodName" ?
This will complement the use cases where method should be called against the whole table, without the need to specify it in additional url parameters.

#45 Re: mORMot 1 » recent changes to TSQLLog » 2013-09-20 08:17:54

I should also note, that I made following change in GarbageCollectorFree in order to avoid AVs:

      if Assigned(PObject(GarbageCollectorFreeAndNilList.List[i])^) then // note: added by chapa
        FreeAndNil(PObject(GarbageCollectorFreeAndNilList.List[i])^);

Edited: And everything seems working fine, no leaks or AVs so far, but will full test again and report if have any problems.

#46 Re: mORMot 1 » recent changes to TSQLLog » 2013-09-20 08:15:47

Try for example 01 - In Memory ORM with ReportMemoryLeaksOnShutdown := True; --> Leaks

I suspect even "uses SynCommons" leaks.
Another example is SynCommons Initialization GarbageCollectorFreeAndNil(SynAnsiConvertList,TObjectList.Create)
Here instance of TObjectList leaks. (not freed on finalization)

Will investigate GarbageCollector, bu I think Ab should have a clue where the problem should be and save me some time smile
Thanks.

Edited: It was my mistake, above initialization and finalization works fine. Sorry

#47 Re: mORMot 1 » recent changes to TSQLLog » 2013-09-20 07:29:42

It does not helped, setting only ptOneFilePerThread.
If I set usual stuff even my application does not shut down. Some finalization block, maybe at some critical section.

I think the problem is deeper, maybe in so called "Garbage Collector". I get other strange random leaks of mORMot created objects, just analyzing some in FastMM leak reports.

#48 mORMot 1 » recent changes to TSQLLog » 2013-09-20 06:53:35

chapa
Replies: 5

Hi ab,

recently I updated the source

I have example declaration in GUI application:

  TMyLog = class (TSQLLog); 

no methods, constructors, destructors.

When shutting down I get AV exception at TSynLog.Destroy when called from GarbageCollectorFree's

FreeAndNil(PObject(GarbageCollectorFreeAndNilList.List[i])^);

on SynCommons finalization.
Debugging shows that

PObject(GarbageCollectorFreeAndNilList[i])^

is nil. Even it is nil, TMyLog leaks, along with other TSQLLog and TObjectList.

It is heavy multithreaded application using the log. If I step through or delay the finalization, many times I do not get the exception.

Do you have any clue what should be the problem or want me to go further with debugging?

Thanks.

#49 Re: mORMot 1 » Synopse.inc incorrectly specifies $WEAKLINKRTTI and $RTTI » 2013-08-26 15:04:40

I confirm it brokes my RTTI also. XE2-XE4 as far as I tried, maybe other versions too.
It has been commented here.

Every time after full BUILD the problem occurs. Then if I modify only one unit and COMPILE, everything runs ok then.

Board footer

Powered by FluxBB