#101 Re: mORMot 1 » Bug: RetrieveBlob with ODBC connection » 2015-08-15 12:31:03

zed

The ColumnDataSize indicates the number of bytes that remain to read, if data is truncated. But previous logic was based on the assumption that it indicates the actual bytes count in the received buffer fColData.

My fix is not optimized and I suggested this as a quick fix. Actually, it works only because Copy function ignore size parameter if it grater then string size.

#102 mORMot 1 » Bug: RetrieveBlob with ODBC connection » 2015-08-15 00:38:28

zed
Replies: 4

If BLOB data is more then 64 bytes, function RetrieveBlob returns data with garbage (part of data + null bytes + another part of data + more nulls etc.).
It seems that bug in function TODBCStatement.ColumnBlob (SynDBODBC.pas). I comment buggy usage of ColumnDataSize and add one line to fix a bug:

function TODBCStatement.ColumnBlob(Col: integer): RawByteString;
var res: TSQLDBStatementGetCol;
    offs: integer;
begin
  res := GetCol(Col,ftBlob);
  case res of
    colNull:      result := '';
    colWrongType: ColumnToTypedValue(Col,ftBlob,result);
    else
    with fColumns[Col] do begin
      result := copy(fColData[Col],1,ColumnDataSize);
      offs := 0;
      while res=colDataTruncated do begin
        //inc(offs,ColumnDataSize);
        res := GetColNextChunk(Col);
        if ColumnDataSize<=0 then
          break;
        //SetLength(result,offs+ColumnDataSize);
        //move(pointer(fColData[Col])^,PByteArray(result)^[offs],ColumnDataSize);
        result := result + copy(fColData[Col],1,ColumnDataSize); // < bug fix >
      end;
    end;
  end;
end;

Bug reproduced on MySQL 5.5 and PostgreSQL 9.3 via ODBC drivers.

#103 Re: mORMot 1 » TTimeLog and Javascript client » 2015-08-11 16:09:03

zed

I just googled that JavaScript bit-level operations are limited to 32 bit. You can use some library to work with 64-bit integers in JS, for example: http://google.github.io/closure-library … _Long.html

#104 Re: mORMot 1 » TTimeLog and Javascript client » 2015-08-11 14:51:35

zed

Take a look to the mORMot sources (SynCommons.pas):

// bits: S=0..5 M=6..11 H=12..16 D=17..21 M=22..25 Y=26..38
// size: S=6 M=6  H=5  D=5  M=4  Y=12
// i.e. S<64 M<64 H<32 D<32 M<16 Y<4096: power of 2 -> use fast shl/shr

procedure TTimeLogBits.From(Y, M, D, HH, MM, SS: cardinal);
begin
  inc(HH,D shl 5+M shl 10+Y shl 14-(1 shl 5+1 shl 10));
  Value := SS+MM shl 6+Int64(HH) shl 12;
end;

#105 Re: mORMot 1 » Bug: MongoDB + UpdateField » 2015-08-04 09:42:04

zed

Why I should use different strings in Update() and UpdateField() methods? I think that behaviour in this methods must be the same.
Besides that, with SQLite3 or DBMS, UpdateField() works fine, error only with MongoDB, and if I wish use the same code with SQLite3 and MongoDB, I should use \\ or \ depends on the DB backend? I think that it's bad idea.

In any case, I decided not to use UpdateField() method in my code anymore.

#106 Re: mORMot 1 » Bug: MongoDB + UpdateField » 2015-08-03 18:53:25

zed

New bug: if we try update text field with "\" symbol it's erase this symbol:

VStr := StringToUTF8('Test string\with slash\with 2 slashes');
VRes := VClientDB.UpdateField(TTestDoc, VDoc.ID, 'Field1', [VStr]);
TSQLRestServerDB(023D6D40).URI(PUT root/TestDoc?setname=Field1&set=%22Test+string%5Cwith+slash%5Cwith+2+slashes%22&wherename=RowID&where=1 inlen=0)
{collection:"test.TestDoc",opCode:"opUpdate",requestID:7,selector:{_id:1},update:{$set:{Field1:"Test stringwith slashwith 2 slashes"}}}

Test code:

uses
  mORMot,
  mORMotSQLite3,
  mORMotMongoDB,
  SynMongoDB,
  SynSQLite3Static,
  SynCommons;

type
  TTestDoc = class(TSQLRecord)
  private
    FField1: RawUTF8;
    FField2: Int64;
  published
    property Field1: RawUTF8 read FField1 write FField1;
    property Field2: Int64 read FField2 write FField2;
  end;

procedure TestUpdateField;
var
  VRes: Boolean;
  VStr: RawUTF8;
  VDoc: TTestDoc;
  VModel: TSQLModel;
  VClient: TMongoClient;
  VDatabase: TMongoDatabase;
  VClientDB: TSQLRestClientDB;
begin
  VClient := TMongoClient.Create('127.0.0.1', 27017);
  try
    VDatabase := VClient.Open('test');
    VModel := TSQLModel.Create([TTestDoc]);
    try
      VClientDB := TSQLRestClientDB.Create(VModel, nil, ':memory:', TSQLRestServerDB);
      try
        if not StaticMongoDBRegisterAll(VClientDB.Server, VDatabase) then begin
          Exit;
        end;

        VClientDB.Server.CreateMissingTables;

        VDoc := TTestDoc.Create;
        try
          VStr := StringToUTF8('Test string\with slash');

          VDoc.FField1 := VStr;
          VDoc.FField2 := 123456;

          if not (VClientDB.Add(VDoc, True) > 0) then begin
            Assert(False);
            Exit;
          end;

          VStr := StringToUTF8('Test string\with slash\with 2 slashes');
          VRes := VClientDB.UpdateField(TTestDoc, VDoc.ID, 'Field1', [VStr]);

          Assert(VRes);

          VRes := VClientDB.Retrieve(VDoc.ID, VDoc);

          Assert(VRes);

          Assert(VDoc.FField1 = VStr);
          Assert(VDoc.FField2 = 123456);
        finally
          VDoc.Free;
        end;
      finally
        VClientDB.Free;
      end;
    finally
      VModel.Free;
    end;
  finally
    VClient.Free;
  end;
end;

In same time, method TSQLRestClient.Update() is save correct text with all slashes.

#107 mORMot 1 » Bug: MongoDB + UpdateField » 2015-07-28 19:33:31

zed
Replies: 4

With MongoDB as a backend, TSQLRestClient.UpdateField() generate bad request and always replace whole document:

{collection:"MyDB.MyColl",opCode:"opUpdate",requestID:41,selector:{_id:1},update:{MyField:1}}

Correct request should use $set keyword:

{collection:"MyDB.MyColl",opCode:"opUpdate",requestID:41,selector:{_id:1},update:{$set:{MyField:true}}}

Please, fix it.

#109 Re: mORMot 1 » Using official MongoDB Connection String » 2015-07-18 20:04:38

zed

I test it with mongod 2.6, 3.0.4 and with MongoLab and it works! ab, you’re awesome.

Auto-detect authentication mechanism by server version is a good idea, but to be on the safe side, сan you make option to use MONGODB-CR with MongoDB 3.xx?

#110 Re: mORMot 1 » Using official MongoDB Connection String » 2015-07-17 21:13:54

zed

Bad news: MongoDB 3.x has 5 (!) auth mechanisms, and MONGODB-CR that I implement is deprecated. Now default is SCRAM-SHA-1 and MongoLab use it.

I found implementations of all mechanisms in one place in PyMongo driver: auth.py and SCRAM is terrible.

#111 Re: mORMot 1 » Using official MongoDB Connection String » 2015-07-17 10:40:16

zed

Here is implementation of TMongoClient.OpenAuth method:

uses
SynCrypto;
...
function TMongoClient.OpenAuth(const DatabaseName, UserName,
  PassWord: RawUTF8): TMongoDatabase;
var res,bson: variant;
    err,nonce,key: RawUTF8;
begin
  if (self=nil) or (UserName='') or (PassWord='') then
    result := nil else begin
    result := TMongoDatabase(fDatabases.GetObjectByName(DatabaseName));
    if result=nil then begin // not already opened -> try now from primary host
      if not fConnections[0].Opened then begin
        fConnections[0].Open;
        // step 1
        bson := BSONVariant('{getnonce:1}');
        err := fConnections[0].RunCommand(DatabaseName,bson,res);
        if err<>'' then
          raise EMongoException.CreateUTF8('%.OpenAuth("%") step1 error: %',[self,DatabaseName,err]);
        // step 2
        nonce := _Safe(res)^.GetValueOrRaiseException('nonce');
        key := MD5(nonce + UserName + MD5(UserName + ':mongo:' + PassWord));
        bson := BSONVariant('{authenticate:1,user:?,nonce:?,key:?}',[],[UserName,nonce,key]);
        err := fConnections[0].RunCommand(DatabaseName,bson,res);
        if err<>'' then
          raise EMongoException.CreateUTF8('%.OpenAuth("%") step2 error: %',[self,DatabaseName,err]);
      end;
      result := TMongoDatabase.Create(Self,DatabaseName);
      fDatabases.AddObject(DatabaseName,result);
    end;
  end;
  if result=nil then
    raise EMongoException.CreateUTF8('%.Open("%") unknown DB',[self,DatabaseName]);
end;

Tested on localhost with MongoDB 2.6 and Delphi 2007.

Based on MongoDB docs: Implement Authentication in a Driver

#112 Re: mORMot 1 » free objects on the server side » 2015-07-10 08:39:32

zed

Simple call aMessarts.Free after use?

#113 mORMot 1 » TDynArray and insert info sorted array » 2015-07-09 11:29:21

zed
Replies: 1

If I need insert item into sorted array, I can do it like this:

  VDynArr.Add(I);
  VDynArr.Sort;

but I think that it's not a very optimized and I suggest to do it differently:

 if not VDynArr.FastLocateSorted(I, VNewIndex) then begin
      VDynArr.Insert(VNewIndex, I);
 end;

After this Insert operation, array will still be sorted.

Implementation of FastLocateSorted is (tested on Delphi 2007):

type
  TDynArrayExt = object(TDynArray)
  public
    function FastLocateSorted(const Elem; out Index: Integer): boolean;
  end;

function TDynArrayExt.FastLocateSorted(const Elem; out Index: Integer): boolean;
var n, L, i, cmp: integer;
    P: PAnsiChar;
begin
  result := False;
  Index := -1; // return -1 if not Sorted
  if fSorted and (@fCompare<>nil) then begin
    P := fValue^;
    L := 0;
    n := Count - 1;
    while L <= n do begin
      i := (L+n) shr 1;
      cmp := fCompare(P[cardinal(i)*ElemSize],Elem);
      if cmp=0 then begin
        L := i;
        result := True;
        Break;
      end else
        if cmp<0 then
          L := i+1 else
          n := i-1;
    end;
    // if result is True - return the index of existing Elem
    // if result is False - return the index where to insert
    Index := L;
  end;
end;

...and small test:

procedure Test;
var
  I, VIndex, VNewIndex, VCount: Integer;
  VIntArr: TIntegerDynArray;
  VDynArr: TDynArrayExt;
begin
  VDynArr.InitSpecific(TypeInfo(TIntegerDynArray), VIntArr, djInteger, @VCount);

  for I := 1 to 6 do begin
    VDynArr.Add(I);
  end;
  VDynArr.Sorted := True;

  I := 3;
  VIndex := VDynArr.Find(I);

  if VIndex >= 0 then begin
    VDynArr.Delete(VIndex);
    VDynArr.Sorted := True;

    if not VDynArr.FastLocateSorted(I, VNewIndex) then begin
      Assert(VNewIndex >= 0);
      Assert(VNewIndex = VIndex);

      VDynArr.Insert(VNewIndex, I);
      VDynArr.Sorted := True;
    end;
  end;
end;

I think it will be useful to see this functional in TDynArray. Or maybe you know other way to do it?

#115 Re: mORMot 1 » JSONtoObject and j2oIgnoreUnknownProperty: odd behavior » 2015-03-16 05:14:49

zed

BBackSoon
Try this: https://gist.github.com/

Gist is a simple way to share snippets and pastes with others. All gists are Git repositories, so they are automatically versioned, forkable and usable from Git.

#117 mORMot 1 » SynLog: Problem with EchoToConsole » 2015-03-08 19:01:24

zed
Replies: 2

Hi. It seems, that "EchoToConsole" not work for me:

program SynLogTest;

{$APPTYPE CONSOLE}

uses
  SysUtils, SynLog, SynCommons;

procedure DoTest;
begin
  with TSynLog.Family do begin
    Level := LOG_VERBOSE;
    EchoToConsole := LOG_VERBOSE;
  end;
  TSynLog.Add.Log(sllDebug, StringToUTF8('Test message'));
  TSynLog.Add.Flush(True);
end;

begin
  try
    Writeln(SynCommons.SYNOPSE_FRAMEWORK_FULLVERSION); // 1.18.1048 FTS3
    DoTest;
    Readln;
  except
    on E:Exception do
      Writeln(E.Classname, ': ', E.Message);
  end;
end.

Log file is created and contains test message, but console output is empty.

Tested on Delphi 2007 and XE2.

#119 mORMot 1 » ObjectToJSON don't work with Delphi2007 and "string" properties » 2015-02-27 00:00:38

zed
Replies: 6

Hi, I try save some object to JSON, like this (Delphi 2007):

type
  TMyClass = class
  private
    FText: string;
  published
    property Text: string read FText write FText;
  end;

  ...

var
  m: TMyClass;
  s: RawUTF8;
begin
  m := TMyClass.Create;
  m.Text := 'One-Один'; // chars in win-1251 encoding (Russian)

  s := ObjectToJson(m, [woStoreClassName]);
  ...
end;

and result is:

{"ClassName":"TMyClass","Text":"One-"}

all Russian chars just disappeared!

Then I take a look to the mORMot sources and found that in string conversions (in my case string type is tkLString) used AnsiStringCodePage property. And this CodePage is always 65535. I talk about this part of code in mORMot.pas:

         procedure TJSONSerializer.WriteObject(Value: TObject; Options: TTextWriterWriteObjectOptions);
        
        ...
        
        {$ifdef FPC}tkAString,{$endif} tkLString: begin
          HR(P);
          Add('"');
          P^.GetLongStrProp(Value,tmp);          
          AddAnyAnsiString(tmp,twJSONEscape,P^.PropType^.AnsiStringCodePage); // AnsiStringCodePage = 65535 ($FFFF)
          Add('"');
        end;

I think that problem in TSynAnsiConvert.Engine function (SynCommons.pas). When I modify it and set to return CurrentAnsiConvert if CodePage is 65535, all my code starts work as expected.

        class function TSynAnsiConvert.Engine(aCodePage: cardinal): TSynAnsiConvert;
        var i: integer;
        begin
          if SynAnsiConvertList=nil then begin
            GarbageCollectorFreeAndNil(SynAnsiConvertList,TObjectList.Create);
            CurrentAnsiConvert := TSynAnsiConvert.Engine(GetACP);
            WinAnsiConvert := TSynAnsiConvert.Engine(CODEPAGE_US) as TSynAnsiFixedWidth;
            UTF8AnsiConvert := TSynAnsiConvert.Engine(CP_UTF8) as TSynAnsiUTF8;
          end;
 --      if aCodePage=0 then begin
 ++      if (aCodePage=0) or (aCodePage=65535) then begin
            result := CurrentAnsiConvert;
            exit;
          end;
          with SynAnsiConvertList do
            for i := 0 to Count-1 do begin
              result := List[i];
              if result.CodePage=aCodePage then
                exit;
            end;
          if aCodePage=CP_UTF8 then
            result := TSynAnsiUTF8.Create(CP_UTF8) else
          if aCodePage=CP_UTF16 then
            result := TSynAnsiUTF16.Create(CP_UTF16) else
          if IsFixedWidthCodePage(aCodePage) then
            result := TSynAnsiFixedWidth.Create(aCodePage) else
            result := TSynAnsiConvert.Create(aCodePage);
          SynAnsiConvertList.Add(result);
        end;

What do you think?

Board footer

Powered by FluxBB