mORMot Open Source

Codorna

Member

Registered: 2018-06-26

Posts: 21

Access Violation when running a query with a huge ammount of data

I've gotta run a query that has a lot of results, millions, ~70 columns, it's all on sqlite.

res := fProps.ExecuteInlined(aSQL,aExpectResults);

the AV ends up happening there:

procedure Movex64; // A. Bouchez' version
asm .noframe // rcx=Source, rdx=Dest, r8=Count
        mov     rax, r8
        sub     rcx, rdx
        je      @11
        jnc     @03
        add     rax, rcx
        jc      @17
@03:    cmp     r8, 8
        jl      @09
        test    dl, 07H
        jz      @06
        test    dl, 01H
        jz      @04
        mov     al, byte ptr[rcx + rdx]
        dec     r8
        mov     byte ptr[rdx], al
        add     rdx, 1
@04:    test    dl, 02H
        jz      @05
        mov     ax, word ptr[rcx + rdx]
        sub     r8, 2
        mov     word ptr[rdx], ax   <---- this line

before AVing the software allocates ~2 gb of RAM

Codorna

Member

Registered: 2018-06-26

Posts: 21

Re: Access Violation when running a query with a huge ammount of data

This keeps happening, when querying something over ~2gb I get an AV

raised exception class $C0000005 with message 'c0000005 ACCESS_VIOLATION'.

if there's less data, same code works.

it's just a:

json := fServer.RetrieveListJSON(fClass, StringToUTF8(fFormatSQLWhere), fBoundsSQLWhere);

fServer being a TSQLRestServerDB

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,544

Website

Re: Access Violation when running a query with a huge ammount of data

Are you on x64? Delphi or FPC?

Codorna

Member

Registered: 2018-06-26

Posts: 21

Re: Access Violation when running a query with a huge ammount of data

Delphi, x64 , i'd get "Out of Memory" on x32

delphi 10.3 now, but I also tried using delphi 10.2

Last edited by Codorna (2019-11-21 19:51:04)

Codorna

Member

Registered: 2018-06-26

Posts: 21

Re: Access Violation when running a query with a huge ammount of data

error happens on TTextWriter.AddString at the

    {$ifdef FPC}Move{$else}MoveFast{$endif}(pointer(Text)^,B[1],L);

line

oz

Member

Registered: 2015-09-02

Posts: 95

Re: Access Violation when running a query with a huge ammount of data

There‘s simply not enough memory available for such an amount of data in X32. You could switch to X64, but this sounds like a design issue to me. Dou you really have to fetch all that data at once?

Codorna

Member

Registered: 2018-06-26

Posts: 21

Re: Access Violation when running a query with a huge ammount of data

It is on x64, it'd be tons of work to go around that, I often do need more than that in memory

Codorna

Member

Registered: 2018-06-26

Posts: 21

Re: Access Violation when running a query with a huge ammount of data

So, apparently strings in delphi are limited to 2GB even on 64bits, any idea what could be done?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,242

Website

Re: Access Violation when running a query with a huge ammount of data

Yes, for TStrRec string type header, with have on Delphi:

length: Longint;           

So limited to 32-bit on all platforms (even Win64).

But on FPC:

length: SizeInt;           

So don't get so many result (use paging)... or switch to FPC which uses SizeInt so 64-bit under Win64/Linux64.