You are not logged in.
Pages: 1
test code, use mormot sample, 02 - Embedded SQLite3 ORM
program Project02;
uses
Forms,
SysUtils,
mORMot,
mORMotSQLite3, SynSQLite3Static,
Unit1 in '..\01 - In Memory ORM\Unit1.pas' {Form1},
SampleData in '..\01 - In Memory ORM\SampleData.pas';
{$R *.res}
begin
Application.Initialize;
Application.CreateForm(TForm1, Form1);
Form1.Caption := ' Sample 02 - Embedded SQLite3 ORM';
Form1.Database := TSQLRestServerDB.Create(Form1.Model,
ChangeFileExt(paramstr(0),'.db3'));
TSQLRestServerDB(Form1.Database).CreateMissingTables(0);
TSQLRestServerDB(Form1.Database).DB.UseCache:=false; //add this line to disable cache
Application.Run;
end.
write ok, but read av occurred
I certainly not make assumption, I use XE to trace here. fFindLastAddedIndex is not accessible. assigned value to fFindLastAddedIndex do get AV.
this problem just occurred by set Usecache=False. maybe late I can give a code to test it.
anyway, here wish you happy new year!
Happy new year to erverybody here!
Sorry, I don't know we can call the method of class which is nil. Usally, it's not safe. Thank you for knowing this.
But here, do have the problem, you assigned value to fFindLastAddedIndex which get av.
the line
fFindLastAddedIndex := -1;
shall be removed.
Please ref
function TSQLDataBase.LockJSON(const aSQL: RawUTF8; aResultCount: PPtrInt): RawUTF8;
begin
if self=nil then
exit; // avoid GPF in case of call from a static-only server
EnterCriticalSection(fLock); // cache access is also protected by fLock
try
if isSelect(pointer(aSQL)) then begin
result := fCache.Find(aSQL,aResultCount); // try to get JSON result from cache
if result<>'' then begin
fCache not checked for nil
What I mean is Cache in server side, the class of TSQLDataBase, which has a public property UseCache
TSQLDataBase = class
......
/// if this property is set, all ExecuteJSON() responses will be cached
// - cache is flushed on any write access to the DB (any not SELECT statement)
// - cache is consistent only if ExecuteJSON() Expand parameter is constant
// - cache is used by TSQLDataBase.ExecuteJSON() and TSQLTableDB.Create()
property UseCache: boolean read GetUseCache write SetUseCache;
The class seems not check the fCache=nil, so, cannot set this property to false.
hi,
When I want to make some test, duto cache, the time is varied. Try to set Server UseCache to False failed. It seems not ready for disable cache.
Is there a plan to make this feature ready?
hi,
About error message, as I mentioned on #1, is not correct displayed for Chinese. though that isn't real problem, but for debug, is not handy.
after I check your code
procedure TSQLDBOracleLib.HandleError(Status: Integer; ErrorHandle: POCIError;
InfoRaiseException: Boolean; LogLevelNoRaise: TSynLogInfo);
begin
......
raise ESQLDBOracle.Create(String(msg));
end;
I don't know how to cast it use your ansiconvert, but can use delphi Tencodeing to convert it correctly.
code as:
raise ESQLDBOracle.Create(TEncoding.UTF8.GetString(Tbytes(msg)));
Now, I don't know its real means. you may know that to integrate the code.
Good done! That's right now.
Thank you very much!
I use tsqlrecord.create(aclient, 'key=?',['In'])function, which the value do bind to SQL.
when I use Binary for text column format of sqlite3. before create db, I do set
iModel.SetCustomCollationForAllRawUTF8('BINARY');
but it seems that sql cache is not aware of that. it's still use HashAnsiStringI for lookup.
so, first retrieve value 'In', which is not in DB, it return null, that's right;
second to retrieve value 'in', which is in DB, it still return null, that's wrong, because json cache in TSQLDataBase.LockJSON.
the cache is not casesensitive there.
Hi,
I try to implement TsqlRecord descendant which need use some aggregate function, i.e. Get Max(ID). After search the forum, I know I can use ExecuteList to Retrieve the data. But, in this thread http://synopse.info/forum/viewtopic.php?id=1099, ab said,
"More generally, aggregate functions are not available - they are not REST-ready, I suspect. "
The aggregate functions is offen used. Here, I have a idea may resolve this problem simply, but I do'nt know if it's possible to implement it.
The idea is to use view to Retrieve such data, so it mean we need to implement a base class which may call TSQLRecordView. This class can only operate with GET, and have a property to define the view creation.
Now, we may use TSQLRecord to bind some view created in server manually. Here, the problem is how to define view creation automatically.
How about that? Is't possible?
Now, it's OK.
I have tested with my DB 11g with charset AL32UTF8 of CHAR/VARCHAR2 and national charset AL16UTF16 of NCHAR/NVARCHAR2.
Thanks, Good worK!
thanks for the quick code-fixed.
I have test the new code about SynDBOracle. Still have a problem here.
You did not process the UTF-8 compatible charset. AL32UTF8 charset is a superset of UTF8. it's UTF8 of Oracle.
on http://docs.oracle.com/cd/B28359_01/ser … m#NLSPG608
they said:
AL32UTF8
An Oracle Database character set for the SQL CHAR data type, which is used for the database character set. It encodes Unicode data in the UTF-8 encoding.
But Its oracle charset ID is not in the list you refer: http://www.mydul.net/charsets.html
folowing code will retrival all the charset ID list of oracle
select nls_charset_id(value) nls_charset_id, value nls_charset_name, to_char(nls_charset_id(value),'xxxx') hex_id
from v$nls_valid_values
where parameter = 'CHARACTERSET'
order by nls_charset_id(value)
the list of My oracle 11g here
...
"871","UTF8"," 367"
"872","UTFE"," 368"
"873","AL32UTF8"," 369"
...
Unfortunately, UTF8 ID <> AL32UTF8 ID, so you shall make modified to code.
The Simple solution may modify TSQLDBOracleConnection.STRToUTF8 such as
const OCI_AL32UTF8=$369
procedure TSQLDBOracleConnection.STRToUTF8(P: PAnsiChar; var result: RawUTF8;
ColumnDBCharSet,ColumnDBForm: integer);
var L: integer;
begin
L := StrLen(PUTF8Char(P));
if (L=0) or (ColumnDBCharSet in [OCI_UTF8, OCI_AL32UTF8]) or (ColumnDBForm=SQLCS_NCHAR) or
(fOCICharSet in [OCI_UTF8, OCI_AL32UTF8]) then
SetString(result,P,L) else
result := fAnsiConvert.AnsiBufferToRawUTF8(P,L);
end;
UTF-8(unicode 3.0) was introduced before Oracle 9i. AL32UTF8(unicode 5.0) is introduced by 9i.
UTFE(unicode 5.0) is used only on EBCDIC platforms.
In my opnion, there is no CreateWithCodePage/CreateWithCharSet needed. when creating connection to OCI, just query Oracle for what Charset is used, this may return charset string such as AL32UTF8/KO16MSWIN949/ZHS16GBK, after that you may use OCINlsCharSetNameToId or a sql query to get related charset_ID;
sql to get DB Charset:
SELECT PROPERTY_VALUE
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME = 'NLS_CHARACTERSET'
sql to convert charsetID
SELECT NLS_CHARSET_ID('AL32UTF8') FROM dual
'AL32UTF8' here is the charset string to query.
but for this auto solution, I don't know if it's available in old db system such as Oracle DB 8.
This time, I use mORMot to process some data which charset is Chinese. My Env is Os_Win7_codepage_936/Oracle_11g_charset_AL32UTF8.
I create prop with default codepage param 0, the result is mess character returned. After a few hour of tracing code, finally I change the codepage to cp_utf8 to make my work done. But the problem here still exists.
Follow the code in unit SynDBOracle.pas, when I use default param 0 to create prop, the Class use GetACP to obtain the OS default codepage 936 first. When query data, convert data returned by OCI from codepage 936 to UTF8. So, I use CP_UTF8 to bypass this convertion. From here, I think SynDBOracle may have a wrong concept about codepage and charset.
I have used Oracle since Oracle8. Each time, charset give us a little trouble when starting a new project, because Chinese Charecter is involved mainly. codepage , charset and NLS_LANG make us mad.
To make clear about problem, I'd like to discuss the concept about codepage , charset, which may make things right.
I think, the codepage and NLS_LANG is related to the client tools of Oracle, such as SQL*PLUS, EXP/IMP; but for direct connect, only charset is necessary to concern. Because, when mORMot connect to Oracle DB using OCI, the client means mORMot, the charset used by client means UTF8. If DB use UTF8, no conversion is needed. If DB use the charset other than UTF8, the conversion is needed. so here mORMot is nothing related to codepage concept. codepage is related to OS or App.
So, Based on the viewpoint above, CreateWithCodePage should changed to CreateWithCharset; the aCodepage param should change to aCharset. for default charset string, we can query DB to get the charset used by DB, or just set it as 'UTF8' simply.
I do'nt know what descripted here is make things clear, because English isn't my native language.
For Programming with Unicode, oracle have the doc here
http://docs.oracle.com/cd/B28359_01/ser … nicode.htm
By the way, another problem is the error msg returned by OCI display not right, I donot know how mORMot convert this message.
I know all the things discussed here not related to ASCII, because all the charset is superset of ASCII. for the guy not use CJK charset, it is difficult to find such error.
And also codepage/charset list in SynDBOracle may not be right for codepage 936. for codepage, consider following table
Common IANA character set Name vs. Oracle character set name
(Http header & html meta tag) (NLS_LANG character set part)
UTF-8 UTF8
windows-1250 EE8MSWIN1250
windows-1251 CL8MSWIN1251
windows-1252 WE8MSWIN1252
windows-1253 EL8MSWIN1253
windows-1254 TR8MSWIN1254
windows-1255 IW8MSWIN1255
windows-1256 AR8MSWIN1256
windows-1257 BLT8MSWIN1257
windows-1258 VN8MSWIN1258
windows-936 or GBK ZHS16GBK
Big5 ZHT16MSWIN950
Big5-HKSCS ZHT16HKSCS – Hong Kong extension of big5
TIS-620 TH8TISASCII
Shift_JIS JA16SJIS
korean or KS_C_5601-1989 KO16MSWIN949
Thank you for your kind reply.
To konw such weird behavior, we can avoid similar situation. for mORMot, we have many solution here. using TSQLDBStatement, or do that as the code you show.
Anyway, Thank you again.
after digging code, this error is not related to mORMot but delphi XE compiler. following is delphi code generated:
smain.pas.47: Rows:= fProps.Execute(_sql,[],nil);
0056BBA6 6AFF push $ff
0056BBA8 6A00 push $00
0056BBAA 8D45E4 lea eax,[ebp-$1c]
0056BBAD 50 push eax
0056BBAE 8D4DE4 lea ecx,[ebp-$1c]
0056BBB1 8B55F4 mov edx,[ebp-$0c]
0056BBB4 8B45F8 mov eax,[ebp-$08]
0056BBB7 E89C96FEFF call TSQLDBConnectionProperties.Execute
0056BBBC 8B55E4 mov edx,[ebp-$1c]
0056BBBF 8D45F0 lea eax,[ebp-$10]
0056BBC2 E861EFE9FF call @IntfCopy
0056BBC7 EB24 jmp $0056bbed
--------------------------------------------------------
smain.pas.47: Rows:= fProps.Execute(_sql,[1],nil);
0056BBA3 6A00 push $00
0056BBA5 6A00 push $00
0056BBA7 8D45F0 lea eax,[ebp-$10]
0056BBAA 50 push eax
0056BBAB C745E001000000 mov [ebp-$20],$00000001
0056BBB2 C645E400 mov byte ptr [ebp-$1c],$00
0056BBB6 8D4DE0 lea ecx,[ebp-$20]
0056BBB9 8B55F4 mov edx,[ebp-$0c]
0056BBBC 8B45F8 mov eax,[ebp-$08]
0056BBBF E89496FEFF call TSQLDBConnectionProperties.Execute
0056BBC4 EB24 jmp $0056bbea
call fProps.Execute with no parameter in open array will make ISQLDBRows interface refcount increase 1, which will result a AV.
anyone has experient about this? how to resolve?
any feedback is welcome?
Thanks for your replay.
I do debug the code, I think the problem is cache used. AV code found at
if fStatement<>nil then begin
if fUseServerSideStatementCache then
OCI.Check(OCI.StmtRelease(fStatement,fError,nil,0,OCI_DEFAULT),fError) else
OCI.HandleFree(fStatement,OCI_HTYPE_STMT);
usecache property seems readonly, so cannot change
maybe , I will wait you to reproduce the error.
I just test Orcale DB access. I found AV when db free. I use XE/Ora 11g. Here is a simple code.
var fProps: TSQLDBOracleConnectionProperties;
_sql: RAWUTF8;
Rows: ISQLDBRows;
begin
fProps:= TSQLDBOracleConnectionProperties.CreateWithCodePage(fOraTNS,
fOraUser, fOraPass,0);
fProps.ForcedSchemaName:= fOraUser;
try
_sql:='select * from dual';
Rows:= fProps.Execute(_sql,[],nil) ;
With Rows Do
While Step Do
Begin
Memo1.Lines.Add(ColumnString(0));
End;
Rows:=nil;
finally
fProps.Free;
end;
After fProps.Free , AV of OCI in TSQLDBOracleStatement.FreeHandles was fired.
folowing code also have AV problem
fProps:= TSQLDBOracleConnectionProperties.CreateWithCodePage(fOraTNS,
fOraUser, fOraPass,0);
fProps.ForcedSchemaName:= fOraUser;
try
_sql:='select * from dual where rownum=?';
With fProps.Execute(_sql,[1],nil) Do
While Step Do
Begin
Memo1.Lines.Add(ColumnString(0));
End;
finally
fProps.Free;
end;
But the strange thing is Execute() function, the following code works no problem.
try
_sql:='select * from dual where rownum=?';
Rows:= fProps.Execute(_sql,[1],nil) ;
With Rows Do
While Step Do
Begin
Memo1.Lines.Add(ColumnString(0));
End;
Rows:=nil;
finally
fProps.Free;
end;
the param there or empty is differ.
Is there something I missed or a bug?
I have a Class such as TDictionary, they may be one class, but many tables keep same data structure. In framework, it seems generate table name automaticly, one class point to one table.
Is there any way to define many tables with one class?
In detail, I save the trie in the data field, that's why I try to search in the field. Of course, it's may be change into normal way, but the speed may vary.
I need more test.
Thanks for your suggestion.
Thank you very much.
I will give the try, because if I load all data into memory in advance, it's search speed is lightning, faster than any other approch I can implement.
Anyway I am not going to write a general one, it's maybe easier.
Thank you again for you advice.
I use TSynBigTableString to store record as RawByteString by RecordLoad/RecordSave. When I try to search the data, I found Getpointer is very fast, but RecordLoad is slow when record is complicated and associated big dynArray. I know that large data to be moved in memory. Is it possible to use pointer here. If it is possible, could you explain the format of RecordSave here, which will save a lot of time for me to dig the code?
today, when I open the switch On (delphi XE)
ReportMemoryLeaksOnShutdown := DebugHook <> 0;
I found AV at AutoFlushProc in syncommons.pas; this may be due to thread delay-free problem.
I temporaryly add a line to resolve this problem
procedure AutoFlushProc(P: pointer); stdcall; // TThread not needed here
var i: integer;
begin
repeat
Sleep(1000); // thread will awake every second to check of pending data
if SynLogFile=nil then
continue;
inc(AutoFlushSecondElapsed);
if AutoFlushThread=0 then continue; //add this line to avoid av when program terminated
for i := 0 to SynLogFile.Count-1 do
PLS look around this problem.
Yes, You are right. I just want user see the interact messages, something like delphi ide message window. I know this logger is better for server, but it's convinent class, I had tried out for display. later maybe I will change the Textwriter to my own when I have time. Anyway Thanks you again for your great job on syncommon.pas unit and synbigtable.pas unit, which I had used for my project.
Thank you very much for your suggestion!
You are right, I have notice that may be slow when log file grow. I may retrival data for latest in limit size. To use TListBox in virtual mode, it's a good idea. I may also make the log message colorful for each type. Thank you again.
In fact, I had compiled that code in Delphi XE, made a test, that seems Ok except for big log stream.
Great! After study a while, I use this class as my project logger, and I load the log content into a memo to watch the response.
Nice, but is there other problem?
TetLog = class(TSynLog)
strict private
fLastSize: Int64;
protected
procedure CreateLogWriter; override;
public
procedure DupLog(Lines: TStrings);
end;
implementation
{ TetLog }
procedure TetLog.CreateLogWriter;
begin
fLastSize:=0;
fWriter := TJSONWriter.Create(nil,fFamily.BufferSize);
inherited;
fWriter.Stream := fWriterStream;
end;
procedure TetLog.DupLog(Lines: TStrings);
var SavePostion: Int64;
begin
if fWriterStream=nil then Exit;
if fLastSize=fWriterStream.Size then Exit;
Lock;
try
SavePostion:= fWriterStream.Position;
fWriterStream.Position:=0;
Lines.LoadFromStream(fWriterStream);
fWriterStream.Position:= SavePostion;
fLastSize := fWriterStream.Size;
finally
Unlock;
end;
end;
I must say TDynArray is great to process array. Currently I use it to packed data into RawByteString for TBigTableRecord storage.
I found a bug maybe, when pack data string large near 64K block, the error occured.
Test code:
var
I, E: integer;
R: RawByteString;
D: TDynArray;
IA: TIntegerDynArray;
begin
D.Init(TypeInfo(TIntegerDynArray), IA);
for I := 0 to 16256 do
begin
E:= I*5;
D.Add(E);
end;
R:=D.SaveTo;
Memo1.Lines.Add(R);
when Execte, the error found on function
function TDynArray.SaveTo: RawByteString;
var Len: integer;
begin
Len := SaveToLength;
SetString(result,nil,Len);
if Len<>0 then
if SaveTo(pointer(result))-pointer(result)<>Len then
Assert(false);
end;
the pointer Calculate seems not right on block large or near 64k.
I downloaded souce (Synopse OpenSource-8195c840e1d682a6)
compile at delphi xe (win7-64bit chinese edition)
some problem found as follows
1.1 Low level common
-IdemPropName: 1/10 FAILED
-Soundex: 1/29 FAILED
1.4 Cryptographic routines:
-Base64: 2/11,994 FAILED
1.6 Synopse PDF
-TPDFDocumentr: 3/4 FAILED
-TPdfDocumentGDI: 1/3 FAILED
2.1 Basic classes:
-TSQLRecord: 4/42 FAILED
2.2 File Based
Exception ESQLException raised with message unrecognized token "'a??, 1791,1872);"
total assertions failed for all test suits:12/3,097,309
What's the problem about that?
Pages: 1