#1 Re: mORMot 1 » SQLite cache » 2013-12-31 14:28:16

test code, use mormot sample, 02 - Embedded SQLite3 ORM

program Project02;

uses
  Forms,
  SysUtils,
  mORMot,
  mORMotSQLite3, SynSQLite3Static,
  Unit1 in '..\01 - In Memory ORM\Unit1.pas' {Form1},
  SampleData in '..\01 - In Memory ORM\SampleData.pas';

{$R *.res}

begin
  Application.Initialize;
  Application.CreateForm(TForm1, Form1);
  Form1.Caption := ' Sample 02 - Embedded SQLite3 ORM';
  Form1.Database := TSQLRestServerDB.Create(Form1.Model,
    ChangeFileExt(paramstr(0),'.db3'));
  TSQLRestServerDB(Form1.Database).CreateMissingTables(0);
  TSQLRestServerDB(Form1.Database).DB.UseCache:=false;    //add this line to disable cache
  Application.Run;
end.

write ok, but read av occurred

#2 Re: mORMot 1 » SQLite cache » 2013-12-31 14:04:13

I certainly not make assumption, I use XE to trace here.  fFindLastAddedIndex  is not accessible. assigned value to fFindLastAddedIndex  do get AV.

this problem just occurred by set Usecache=False. maybe late I can give a code to test it.

anyway, here wish you happy new year!

Happy new year to erverybody here!

#3 Re: mORMot 1 » SQLite cache » 2013-12-31 02:42:46

Sorry, I don't know we can call the method of class which is nil. Usally, it's not safe. Thank you for knowing this.

But here, do have the problem, you assigned value to fFindLastAddedIndex  which get av.
the line

fFindLastAddedIndex := -1;

shall be removed.

#4 Re: mORMot 1 » SQLite cache » 2013-12-30 09:57:58

Please ref

function TSQLDataBase.LockJSON(const aSQL: RawUTF8; aResultCount: PPtrInt): RawUTF8;
begin
  if self=nil then
    exit; // avoid GPF in case of call from a static-only server
  EnterCriticalSection(fLock); // cache access is also protected by fLock
  try
    if isSelect(pointer(aSQL)) then begin
      result := fCache.Find(aSQL,aResultCount); // try to get JSON result from cache
      if result<>'' then begin

fCache not checked for nil

#5 Re: mORMot 1 » SQLite cache » 2013-12-30 00:23:36

What I mean is Cache in server side, the class of TSQLDataBase, which has a public property UseCache

  TSQLDataBase = class
   ......
    /// if this property is set, all ExecuteJSON() responses will be cached
    // - cache is flushed on any write access to the DB (any not SELECT statement)
    // - cache is consistent only if ExecuteJSON() Expand parameter is constant
    // - cache is used by TSQLDataBase.ExecuteJSON() and TSQLTableDB.Create()
    property UseCache: boolean read GetUseCache write SetUseCache;

   
   The class seems not check the fCache=nil, so, cannot set this property to false.

#6 mORMot 1 » SQLite cache » 2013-12-29 01:34:37

richard6688
Replies: 10

hi,
  When I want to make some test, duto cache, the time is varied. Try to set Server UseCache to False failed. It seems not ready for disable cache.
  Is there a plan to make this feature ready?

#7 Re: mORMot 1 » Oracle Charset and codepage » 2013-12-18 03:58:49

hi,
   About error message, as I mentioned on #1, is not correct displayed for Chinese. though that isn't real problem, but for debug, is not handy.
   after I check your code

procedure TSQLDBOracleLib.HandleError(Status: Integer; ErrorHandle: POCIError;
  InfoRaiseException: Boolean; LogLevelNoRaise: TSynLogInfo);
begin
......
 raise ESQLDBOracle.Create(String(msg));
end;

I don't know how to cast it use your ansiconvert, but can use delphi Tencodeing to convert it correctly.
code as:

  raise ESQLDBOracle.Create(TEncoding.UTF8.GetString(Tbytes(msg)));

Now, I don't know its real means. you may know that to integrate the code.

#9 Re: mORMot 1 » Bug in sql cache for sqlite SetCustomCollationForAllRawUTF8 » 2013-12-15 15:21:27

I use tsqlrecord.create(aclient, 'key=?',['In'])function, which the value do bind to SQL.

#10 mORMot 1 » Bug in sql cache for sqlite SetCustomCollationForAllRawUTF8 » 2013-12-15 14:22:15

richard6688
Replies: 4

when I use Binary for text column format of sqlite3. before create db, I do set
iModel.SetCustomCollationForAllRawUTF8('BINARY');

but it seems that sql cache is not aware of that. it's still use HashAnsiStringI for lookup.
so, first retrieve value 'In', which is not in DB, it return null, that's right;
second to retrieve value 'in', which is in DB, it still return null, that's wrong, because json cache in TSQLDataBase.LockJSON.
the cache is not casesensitive there.

#11 mORMot 1 » TSQLRecord & aggregate function » 2013-12-13 13:53:13

richard6688
Replies: 1

Hi,
   I try  to implement TsqlRecord descendant which need use some aggregate function, i.e. Get Max(ID). After search the forum, I know I can use ExecuteList to Retrieve the data. But, in this thread http://synopse.info/forum/viewtopic.php?id=1099, ab said,
"More generally, aggregate functions are not available - they are not REST-ready, I suspect. "

The aggregate functions is offen used. Here, I have a idea may resolve this problem simply, but I do'nt know if it's possible to implement it.
The idea is to use view to Retrieve such data, so it mean we need to implement a base class which may call TSQLRecordView. This class can only operate with GET, and have a property to define the view creation.

Now, we may use TSQLRecord to bind some view created in server manually. Here, the problem is how to define view creation automatically.

How about that? Is't possible?

#12 Re: mORMot 1 » Oracle Charset and codepage » 2013-12-10 10:23:40

Now, it's OK.

I have tested with my DB 11g with charset AL32UTF8 of CHAR/VARCHAR2 and national charset AL16UTF16 of NCHAR/NVARCHAR2.

Thanks, Good worK!

#13 Re: mORMot 1 » Oracle Charset and codepage » 2013-12-10 01:10:38

thanks for the quick code-fixed.

I have test the new code about SynDBOracle. Still have a problem here.
You did not process the UTF-8 compatible charset. AL32UTF8 charset is a superset of UTF8. it's UTF8 of Oracle.
on http://docs.oracle.com/cd/B28359_01/ser … m#NLSPG608
they said:

AL32UTF8

An Oracle Database character set for the SQL CHAR data type, which is used for the database character set. It encodes Unicode data in the UTF-8 encoding.

But Its  oracle charset ID is not in the list you refer: http://www.mydul.net/charsets.html
folowing code will retrival all the charset ID list of oracle

select nls_charset_id(value) nls_charset_id,  value nls_charset_name, to_char(nls_charset_id(value),'xxxx') hex_id
from  v$nls_valid_values
where parameter = 'CHARACTERSET'
order by nls_charset_id(value)

the list of My oracle 11g here

...
"871","UTF8","  367"
"872","UTFE","  368"
"873","AL32UTF8","  369"
...

Unfortunately, UTF8 ID <> AL32UTF8 ID, so you shall make modified to code.
The Simple solution may modify TSQLDBOracleConnection.STRToUTF8 such as

const OCI_AL32UTF8=$369
procedure TSQLDBOracleConnection.STRToUTF8(P: PAnsiChar; var result: RawUTF8;
  ColumnDBCharSet,ColumnDBForm: integer);
var L: integer;
begin
  L := StrLen(PUTF8Char(P));
  if (L=0) or (ColumnDBCharSet in [OCI_UTF8, OCI_AL32UTF8]) or (ColumnDBForm=SQLCS_NCHAR) or
     (fOCICharSet in [OCI_UTF8, OCI_AL32UTF8]) then
    SetString(result,P,L) else
    result := fAnsiConvert.AnsiBufferToRawUTF8(P,L);
end;

UTF-8(unicode 3.0) was introduced before Oracle 9i. AL32UTF8(unicode 5.0) is introduced by 9i.
UTFE(unicode 5.0) is used only on EBCDIC platforms.

#14 Re: mORMot 1 » Oracle Charset and codepage » 2013-12-09 11:57:01

In my opnion, there is no CreateWithCodePage/CreateWithCharSet needed. when creating connection to OCI, just query Oracle for what Charset is used, this may  return charset string such as AL32UTF8/KO16MSWIN949/ZHS16GBK, after that you may use OCINlsCharSetNameToId or a sql query to get related charset_ID;

sql to get DB Charset:

SELECT PROPERTY_VALUE
  FROM DATABASE_PROPERTIES
 WHERE PROPERTY_NAME = 'NLS_CHARACTERSET'

sql to convert charsetID

 SELECT NLS_CHARSET_ID('AL32UTF8') FROM dual

'AL32UTF8' here is the charset string to query.

but for this auto solution, I don't know if it's available in old db system such as Oracle DB 8.

#15 mORMot 1 » Oracle Charset and codepage » 2013-12-09 02:34:13

richard6688
Replies: 8

This time, I use mORMot to process some data which charset is Chinese. My Env is Os_Win7_codepage_936/Oracle_11g_charset_AL32UTF8.
I create prop with default codepage param 0, the result is mess character returned. After a few hour of tracing code, finally I change the codepage to cp_utf8 to make my work done. But the problem here still exists.

Follow the code in unit SynDBOracle.pas, when I use default param 0 to create prop, the Class use GetACP to obtain the OS default codepage 936 first. When query data, convert data returned by OCI from codepage 936 to UTF8. So, I use CP_UTF8 to bypass this convertion. From here, I think SynDBOracle may have a wrong concept about codepage and charset.

I have used Oracle since Oracle8. Each time, charset give us a little trouble when starting a new project, because Chinese Charecter is involved mainly. codepage , charset and NLS_LANG make us mad.

To make clear about problem, I'd like to discuss the concept about codepage , charset, which may make things right.

I think, the codepage and NLS_LANG is related to the client tools of Oracle, such as SQL*PLUS, EXP/IMP; but for direct connect, only charset is necessary to concern. Because, when mORMot connect to Oracle DB using OCI, the client means mORMot, the charset used by client  means UTF8. If DB use UTF8, no conversion is needed. If DB use the charset other than UTF8, the conversion is needed. so here mORMot  is nothing related to codepage concept. codepage is related to OS or App.

So, Based on the viewpoint above, CreateWithCodePage should changed to CreateWithCharset; the aCodepage param should change to aCharset. for default charset string, we can query DB to get the charset used by DB, or just set it as 'UTF8' simply.

I do'nt know what descripted here is make things clear, because English isn't my native language.

For Programming with Unicode, oracle have the doc here
http://docs.oracle.com/cd/B28359_01/ser … nicode.htm

By the way, another problem is the error msg returned by OCI display not right, I donot know how mORMot convert this message.

I know all the things discussed here not related to ASCII, because all the charset is superset of ASCII. for the guy not use CJK charset, it is difficult to find such error.

And also codepage/charset list in SynDBOracle may not be right for codepage 936. for codepage, consider following table

Common IANA character set Name      vs.     Oracle character set name
   (Http header & html meta tag)               (NLS_LANG character set part)
 

    UTF-8                                           UTF8
    windows-1250                                EE8MSWIN1250
    windows-1251                                CL8MSWIN1251
    windows-1252                                WE8MSWIN1252
    windows-1253                                EL8MSWIN1253
    windows-1254                                TR8MSWIN1254
    windows-1255                                IW8MSWIN1255
    windows-1256                                AR8MSWIN1256
    windows-1257                                BLT8MSWIN1257
    windows-1258                                VN8MSWIN1258
    windows-936 or GBK                        ZHS16GBK
    Big5                                            ZHT16MSWIN950
    Big5-HKSCS                                  ZHT16HKSCS – Hong Kong extension of big5
    TIS-620                                       TH8TISASCII
    Shift_JIS                                     JA16SJIS
    korean or KS_C_5601-1989             KO16MSWIN949

#16 Re: mORMot 1 » OCI AV when OracleConnection free » 2013-12-04 23:50:42

Thank you for your kind reply.

To konw such weird behavior, we can avoid similar situation. for mORMot, we have many solution here. using TSQLDBStatement, or do that as the code you show.

Anyway, Thank you again.

#17 Re: mORMot 1 » OCI AV when OracleConnection free » 2013-12-04 02:11:40

after digging code, this error is not related to mORMot but delphi XE compiler. following is delphi code generated:

smain.pas.47: Rows:= fProps.Execute(_sql,[],nil);
0056BBA6 6AFF             push $ff
0056BBA8 6A00             push $00
0056BBAA 8D45E4           lea eax,[ebp-$1c]
0056BBAD 50               push eax
0056BBAE 8D4DE4           lea ecx,[ebp-$1c]
0056BBB1 8B55F4           mov edx,[ebp-$0c]
0056BBB4 8B45F8           mov eax,[ebp-$08]
0056BBB7 E89C96FEFF       call TSQLDBConnectionProperties.Execute
0056BBBC 8B55E4           mov edx,[ebp-$1c]
0056BBBF 8D45F0           lea eax,[ebp-$10]
0056BBC2 E861EFE9FF       call @IntfCopy
0056BBC7 EB24             jmp $0056bbed
--------------------------------------------------------
smain.pas.47: Rows:= fProps.Execute(_sql,[1],nil);
0056BBA3 6A00             push $00
0056BBA5 6A00             push $00
0056BBA7 8D45F0           lea eax,[ebp-$10]
0056BBAA 50               push eax
0056BBAB C745E001000000   mov [ebp-$20],$00000001
0056BBB2 C645E400         mov byte ptr [ebp-$1c],$00
0056BBB6 8D4DE0           lea ecx,[ebp-$20]
0056BBB9 8B55F4           mov edx,[ebp-$0c]
0056BBBC 8B45F8           mov eax,[ebp-$08]
0056BBBF E89496FEFF       call TSQLDBConnectionProperties.Execute
0056BBC4 EB24             jmp $0056bbea

call fProps.Execute with no parameter in open array will make ISQLDBRows interface refcount increase 1, which will result a AV.
anyone has experient about this? how to resolve?

any feedback is welcome?

#18 Re: mORMot 1 » OCI AV when OracleConnection free » 2013-11-26 07:36:01

Thanks for your replay.

I do debug the code, I think the problem is cache used. AV code found at

  if fStatement<>nil then begin
    if fUseServerSideStatementCache then
      OCI.Check(OCI.StmtRelease(fStatement,fError,nil,0,OCI_DEFAULT),fError) else
      OCI.HandleFree(fStatement,OCI_HTYPE_STMT);

usecache property seems readonly, so cannot change
maybe , I will wait you to reproduce the error.

#19 mORMot 1 » OCI AV when OracleConnection free » 2013-11-26 00:35:27

richard6688
Replies: 5

I just test Orcale DB access. I found AV when db free. I use XE/Ora 11g. Here is a simple code.

var fProps: TSQLDBOracleConnectionProperties;
_sql: RAWUTF8;
Rows: ISQLDBRows;
begin
  fProps:= TSQLDBOracleConnectionProperties.CreateWithCodePage(fOraTNS,
    fOraUser, fOraPass,0);
  fProps.ForcedSchemaName:= fOraUser;
  try
    _sql:='select * from dual';
    Rows:= fProps.Execute(_sql,[],nil) ;
    With Rows Do
    While Step Do
    Begin 
      Memo1.Lines.Add(ColumnString(0));
    End; 
    Rows:=nil;
  finally
    fProps.Free;
  end;

After  fProps.Free ,  AV of OCI in TSQLDBOracleStatement.FreeHandles was fired.

folowing code also have AV problem

  fProps:= TSQLDBOracleConnectionProperties.CreateWithCodePage(fOraTNS,
    fOraUser, fOraPass,0);
  fProps.ForcedSchemaName:= fOraUser;
  try
    _sql:='select * from dual where rownum=?';
    With fProps.Execute(_sql,[1],nil) Do
    While Step Do
    Begin 
      Memo1.Lines.Add(ColumnString(0));
    End; 
  finally
    fProps.Free;
  end;

But the strange thing is  Execute() function, the following code works no problem.

  try
    _sql:='select * from dual where rownum=?';
    Rows:= fProps.Execute(_sql,[1],nil) ;
    With Rows Do
    While Step Do
    Begin 
      Memo1.Lines.Add(ColumnString(0));
    End; 
    Rows:=nil;
  finally
    fProps.Free;
  end;

the param there or empty is differ.

Is there something I missed or a bug?

#20 mORMot 1 » TSQLRecord and SQLTableName » 2013-08-21 12:45:21

richard6688
Replies: 1

I have a Class such as TDictionary, they may be one class, but many tables keep same data structure. In framework, it seems generate table name automaticly, one class point to one table.

Is there any way to define many tables with one class?

#21 Re: Source Code repository » SynCommons: TDynArray and Record compare/load/save using fast RTTI » 2012-01-05 01:01:58

In detail, I save the trie in the data field, that's why I try to search in the field. Of course, it's may be change into normal way, but the speed may vary.
I need more test.
Thanks for your suggestion.

#22 Re: Source Code repository » SynCommons: TDynArray and Record compare/load/save using fast RTTI » 2012-01-04 11:46:26

Thank you very much.
I will give the try, because if I load all data into memory in advance, it's search speed is lightning, faster than any other approch I can implement.
Anyway I am not going to write a general one, it's maybe easier.

Thank you again for you advice.

#23 Re: Source Code repository » SynCommons: TDynArray and Record compare/load/save using fast RTTI » 2012-01-04 04:02:15

I use TSynBigTableString to store record as RawByteString by RecordLoad/RecordSave. When I try to search the data, I found Getpointer is very fast, but RecordLoad is slow when record is complicated and associated big dynArray. I know that large data  to be moved in memory.  Is it possible to use pointer here. If it is possible, could you explain the format of RecordSave here, which will save a lot of time for me to dig the code?

#24 Re: Source Code repository » Introducing enhanced logging mechanism » 2011-06-04 07:52:32

today, when I open the switch On (delphi XE)
 
   ReportMemoryLeaksOnShutdown := DebugHook <> 0;

I found AV at AutoFlushProc in syncommons.pas; this may  be due to thread delay-free problem.
I temporaryly add a line to resolve this problem

procedure AutoFlushProc(P: pointer); stdcall;  // TThread not needed here
var i: integer;
begin
  repeat
    Sleep(1000); // thread will awake every second to check of pending data
    if SynLogFile=nil then
      continue;
    inc(AutoFlushSecondElapsed);
    if AutoFlushThread=0 then continue;  //add this line to avoid av when program terminated
    for i := 0 to SynLogFile.Count-1 do


PLS look around this problem.

#25 Re: Source Code repository » Introducing enhanced logging mechanism » 2011-05-13 07:12:39

Yes, You are right. I just want user see the interact messages, something like delphi ide message window. I know this logger is better for server, but it's convinent class, I had tried out for display. later maybe I will change the Textwriter to my own when I have time. Anyway Thanks you again for your great job on syncommon.pas unit and synbigtable.pas unit, which I had used for my project.

#26 Re: Source Code repository » Introducing enhanced logging mechanism » 2011-05-13 01:02:36

Thank you very much for your suggestion!
You are right, I have notice that may be slow when log file grow. I may retrival data for latest in limit size. To use TListBox in virtual mode, it's a good idea. I may also make the log message colorful for each type. Thank you again.
In fact, I had compiled that code in Delphi XE, made a test, that seems Ok except for big log stream.

#27 Re: Source Code repository » Introducing enhanced logging mechanism » 2011-05-12 16:43:10

Great! After study a while, I use this class as my project logger, and I load the log content into a memo to watch the response.
Nice, but is there other problem?

  TetLog = class(TSynLog)
  strict private
    fLastSize: Int64;
  protected
    procedure CreateLogWriter; override;
  public
    procedure DupLog(Lines: TStrings);
  end;

implementation

{ TetLog }

procedure TetLog.CreateLogWriter;
begin
  fLastSize:=0;
  fWriter := TJSONWriter.Create(nil,fFamily.BufferSize);
  inherited;
  fWriter.Stream := fWriterStream;
end;

procedure TetLog.DupLog(Lines: TStrings);
var SavePostion: Int64;
begin
  if fWriterStream=nil then Exit;
  if fLastSize=fWriterStream.Size then Exit;
  Lock;
  try
    SavePostion:= fWriterStream.Position;
    fWriterStream.Position:=0;
    Lines.LoadFromStream(fWriterStream);
    fWriterStream.Position:= SavePostion;
    fLastSize := fWriterStream.Size;
  finally
    Unlock;
  end;
end;

#28 Re: Source Code repository » SynCommons: TDynArray and Record compare/load/save using fast RTTI » 2011-04-13 01:34:56

I must say TDynArray is great to process array. Currently I use it to packed data into RawByteString for TBigTableRecord storage.
I found a bug maybe, when pack data string large near 64K block,  the  error occured.
Test code:

var
I, E: integer;
R: RawByteString;
D: TDynArray;
IA: TIntegerDynArray;
begin
  D.Init(TypeInfo(TIntegerDynArray), IA);
  for I := 0 to 16256 do
  begin
    E:= I*5;
    D.Add(E);
  end;
  R:=D.SaveTo;
  Memo1.Lines.Add(R);

when Execte, the error found on function

function TDynArray.SaveTo: RawByteString;
var Len: integer;
begin
  Len := SaveToLength;
  SetString(result,nil,Len);
  if Len<>0 then
    if SaveTo(pointer(result))-pointer(result)<>Len then
      Assert(false);
end;

the pointer Calculate seems not right on block large or near 64k.

#29 mORMot 1 » TestSQl3 problem on delphi XE (win7-64bit) » 2011-04-05 03:03:56

richard6688
Replies: 1

I downloaded souce (Synopse OpenSource-8195c840e1d682a6)
compile at delphi xe (win7-64bit chinese edition)

some problem found as follows

1.1 Low level common
   -IdemPropName: 1/10 FAILED
   -Soundex: 1/29 FAILED

1.4 Cryptographic routines:
   -Base64: 2/11,994 FAILED

1.6 Synopse PDF
    -TPDFDocumentr: 3/4 FAILED
    -TPdfDocumentGDI: 1/3 FAILED

2.1 Basic classes:
   -TSQLRecord: 4/42 FAILED

2.2 File Based
    Exception ESQLException raised with message unrecognized token "'a??, 1791,1872);"


total assertions failed for all test suits:12/3,097,309

What's the problem about that?

Board footer

Powered by FluxBB