mORMot Open Source

pvn0

Member

From: Slovenia

Registered: 2018-02-12

Posts: 211

Re: Fast MM5

thx ab, I will be using fpcx64mm as I gradually move over to mORMot 2.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Fast MM5

For mORMot 1, you have SynFPCx64MM which is almost the same.

Dr. John

Member

Registered: 2022-10-25

Posts: 3

Re: Fast MM5

I think I've discovered a bug.   It's probably associated with an interface to FastMM5.  When I use an array of a typed variable of multiple elements, and then reference an element of a member of that array using,

  with array[index] do
    begin
      .
      .
      .
    end

things don't work.  They work perfectly fine using FPC's built-in memory manager. 

I've used Fast MM5 with other projects (mostly image processing) and I love it, even though it is a bit more picky than FPC's built-in memory manager.  But, this seems to be a significant bug.

If you'd like a copy of the source code that is causing me a problem, then I can post it.  However, I think the description is adequate to start investigating.

Thanks for your great contributions.

Last edited by Dr. John (2022-10-25 22:04:11)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Fast MM5

@DrJohn
I don't understand exactly the problem. Is it with FastMM5 or SynFPCx64MM?
But FastMM5 is not available on FPC.

The "with array[index] do" pattern is a very common one. We use it in several places in mORMot with no problem.
My guess is that inside the "begin .. end" you use a SetLength(array, ...) which passes by chance on FPC builtin MM, because the resize is done in-place and the array[] pointer don't change, whereas with another MM, a new block is allocated, so a GPF occurs.
The problem is likely to be inside this "begin...end" block of your code, not the MM.

Dr. John

Member

Registered: 2022-10-25

Posts: 3

Re: Fast MM5

Thanks ab for getting back to me.  I think the problem is with FASTMM5.  I first discovered FASTMM5 here:

https://blog.synopse.info/?post/2020/05 … 4-assembly

while trying to track down memory management errors for my image processing application.  At that time I got a more recent copy from a link from this forum.  Most recently I picked up an updated version from here:

https://github.com/synopse/mORMot2/tree/master/src/core  (mormot.core.fpcx64mm)

I assume that they are all equal except for revisions to update or bug-fix.  As I said, I had good luck trying to fix the problems I was having with the FPC internal memory manager with this code for my image processing application.  It was much more stable and didn’t crash after rather heavy use.

I tried it on my an update to software that I’ve written for finding duplicate files whereupon I discovered this problem.  I’m using Lazarus v1.82 as my development environment and was interested in trying this memory manager to help speed things up when dealing with millions of files.

I haven’t done extensive testing to try to determine the exact location in the code where it fails, nor how it fails.  However, the logic for determining whether there are duplicate files is in the following code snippet:

      // mark the currently matching files
      no_match := true;
      for count_1 := 0 to number_of_files - 1 do
        with compare_files[count_1] do
          begin
            for count_2 := count_1 + 1 to number_of_files do
              begin
                if files[file_index - count_1].duplicated and
                   files[file_index - count_2].duplicated and
                   (CRC = compare_files[count_2].CRC) then
                  begin
                    matched := true;
                    compare_files[count_2].matched := true;
                    no_match := false;
                  end;
              end;
            // if not currently matched, short circuit future evaluations of this file
            files[file_index - count_1].duplicated := matched;
          end;

It is here that the logic fails with the FASTMM5 memory manager but works fine with the FPC memory manager.  matched and CRC are elements of the type comprising the array compare_files.

I’m happy to assist you in determining what the failure mode is.  However, I wanted to bring it to your attention ASAP so that others might avoid this problem.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Fast MM5

mormot.core.fpcx64mm is not based on FastMM5 but on FastMM4 with some enhancements. So I am a bit confused with your questions.
So I guess you have the trouble with mormot.core.fpcx64mm ?

My first idea is that the MM is not the culprit, but the code logic.
We use mormot.core.fpcx64mm since years on heavily memory intensive projects, with no problem.

At first, try to run the code with the FPC heaptrc unit (check for mem leaks).

Then try to rewrite the code without any "with" but a local variable.
You should not use "with" over two nested loops, it is really confusing.
It is difficult for me to find out what it is supposed to do, because we don't know when a variable is global, and when it is part of the "with" scope.

Finally, try to make a reproducible sample of the issue, or at least publish the whole code.
But on a gitst, not on the forum itself, as stated by the forum rules.

Dr. John

Member

Registered: 2022-10-25

Posts: 3

Re: Fast MM5

Hi ab,

First let me thank you for your help and patience.  Second, let me apologize for the confusion that I’ve created on a number of fronts.  It’s because this project is really an old tool that I created years ago and update from time to time with “improvements.”  Hence, it isn’t really my focus.  I hope to correct the confusion with the following. 

1) Yes, it is really mormot.core.fpcx64mm that I’m using.  The confusion comes from the title of the topic, but most of the conversation seems to be related to fpcx64mm.

2) I had assumed that the problem that I was experiencing was associated with fpcx64mm and that it was related to the with statement as I previously mentioned.  The latter assumption was incorrect.  In all cases that I checked, the with construct works as expected.  However, the logical failure was so close to the code that I previously mentioned, that I thought it obvious that this was where the problem lies.

I still do have a problem with fpcx64mm however.  It has to do with with the following function:

  getheapstatus.TotalFree

I used this to dynamically determine the size of the file buffers I was using.  With fpcx64mm, it always seems to return 0.  With the fpc internal memory manager it returns a sensible number.  What was happening is that my code, running with 0 size file buffers would simply return identical CRC’s for each file.  The rest of the code worked as expected under the circumstances.

The work-around is to set my file buffer size with a fixed number, for now.  However, it would be nice to be able to use the above mentioned call to get an understanding of the available memory on the heap.  I assume that this is something not so difficult to do.

Many thanks for your important contributions to the Pascal world.

Last edited by Dr. John (2022-10-29 19:58:24)