mORMot Open Source

blue

Member

Registered: 2023-05-05

Posts: 9

Utf8ToString does not support Chinese

OS: Windows 11
IDE: delphi 2007

procedure Test;
var
  src, dst                              : RawUtf8;
begin
  src := '中文abc';
  dst := StringToUtf8(src);
  Writeln('StringToUtf8: ', dst, ', src length: ', Length(src), ', dst length: ', Length(dst));

  src := Utf8ToString(dst);
  Writeln('Utf8ToString: ', src, ', src length: ', Length(src), ', dst length: ', Length(dst));
end;

Last edited by blue (2023-05-05 17:00:28)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Utf8ToString does not support Chinese

Your are making a confusion between string and RawUtf8 variables - this is the correct way on Delphi XE for instance:

procedure testunicode;
var
  s1, s2: string;
  dst: RawUtf8;
begin
  s1 := '中文abc';
  dst := StringToUtf8(s1);
  s2 := Utf8ToString(dst);
  if s1 = s2 then
    writeln('ok')
  else
    writeln('ko');
end;

blue

Member

Registered: 2023-05-05

Posts: 9

Re: Utf8ToString does not support Chinese

getting an incorrect result on Delphi 2007.

fixed patch:

function Unicode_GetWideToAnsiSize(W: PWideChar; LW, CodePage: PtrInt): integer;
begin
  result := WideCharToMultiByte(CodePage, 0, W, LW, nil, 0, nil, nil);
end;

function TSynAnsiConvert.UnicodeBufferToAnsi(Dest: PAnsiChar;
  Source: PWideChar; SourceChars: cardinal): PAnsiChar;
..
    // rely on the Operating System for all remaining ASCII characters
    if SourceChars <> 0 then begin
      inc(Dest,
        Unicode_WideToAnsi(Source, Dest, SourceChars, Unicode_GetWideToAnsiSize(Source, SourceChars, fCodePage), fCodePage));

Last edited by blue (2023-05-05 21:33:58)

cybexr

Member

Registered: 2016-09-14

Posts: 81

Re: Utf8ToString does not support Chinese

problem exists with mormot2, tested below:

D7 + mormot

D7 + mormot2

profh

Member

Registered: 2010-07-02

Posts: 161

Re: Utf8ToString does not support Chinese

uses LConvEncoding, and try CP936ToUTF8 or UTF8ToCP936.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Utf8ToString does not support Chinese

Please try https://github.com/synopse/mORMot2/commit/0cb44638

blue

Member

Registered: 2023-05-05

Posts: 9

Re: Utf8ToString does not support Chinese

Please try https://github.com/synopse/mORMot2/commit/0cb44638

test passed.

itSDS

Member

From: Germany

Registered: 2014-04-24

Posts: 516

Re: Utf8ToString does not support Chinese

just for my interest
in the above sample you use Writeln( with dst as parameter
i think writeln uses an implicit cast of RawUtf8 To String or am i wrong ?

---

Rad Studio 12.1 Santorini