mORMot Open Source

Hi

I have some really big files. Usually they stay in 1-6GB range. But sometimes these file can be even up to 14GB large.
Each file has several millions of entries like this (this is a simplified example):

string_about_100_to_1000_chars   a_short_string   number1    number2     number3 ... number10                  <--------- there is no more than 2KB of data on each entry

I need to:
1. display all rows in a TStringGrid (this is easy because I can only load from disk the data associated with the visible rows)
2. to sort the data (when the user click a column header).

My questions are:

  1. can I use Big Table for this
  2. if yes, can you please provide a skeleton example and some basic concepts on how to start (to store my data into the table for 'fast' sorting on computers with 2-4GB RAM)?
  2. will it work also on Lazarus?

Many thanks

Hi.
There is some way (email, forum, twitter, newslist, etc) to find out when a new version of BigTable (or other library) is released?

Hi.
I have a question: When to use tfoIndex, tfoUnique, tfoCaseInsensitive?

Let's say I want to store in the table two strings, some binary data and some integers. Something like this:

fdName := Table.AddField('Name' , tftWinAnsi, [tfoIndex] );
fdBase  := Table.AddField('Base'  , tftWinAnsi, [] );
fdNoOf  := Table.AddField('NoOf'  , tftInt32  , [] );

When I read the table back, I need only few of its columns (the entire column) - I just read the columns in memory and process the data. So I don't do searches in the table. Is it ok is none of the fields have the tfoIndex and tfoUnique attribute set?

Also, if I set to tfoCaseInsensitive attribute, it will work faster (usually CaseInsensitive strings are faster, right)?
Thanks

Hi.
I want to create a table and and to write to it records. Each record contains a string and an integer:

  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1');       
  TRY
    fdBases   := Table.AddField ('Base'    , tftWinAnsi, [tfoIndex] ); 
    fdNoOfBas := Table.AddField( 'NoOfBase', tftInt32  , [tfoIndex, tfoUnique] ); 
    Table.AddFieldUpdate;     

Tomorrow I may add a boolean field also:

  procedure TForm1.WriteExtraFields;
  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1'); 
  TRY
    Table.AddField ('bool', tftBoolean);
    Table.AddFieldUpdate;       

But forget tomorrow. I want to read the table today, while the third field was not yet added. But the program crashes here:

      { Retrieve fields }
      aStr:= ReadRec.GetFieldValue(fdBases);
      aInt:= ReadRec.GetFieldValue(fdNoOfBas);
      bol := ReadRec.GetFieldValue(fdBoolean);                <-------- Crash here.

How do I test if a field exists before reading it. There is something like: "if Table.FieldExists then DoStuff" ?

----------------------
EDIT:

I also have problems updating the current records when adding the new column:

procedure TForm1.WriteExtraFields;
VAR
   i: Integer;
   Table: TSynBigTableRecord;
   CurRec: TSynTableData;
   NewField: TSynTableFieldProperties;
begin
  { Create table }
  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1');
  TRY
    { Define an extra fields/column to be added to the table }
    NewField:= Table.AddField ('bool', tftBoolean);
    Table.AddFieldUpdate;

    for i:= 1 to 10 DO
     begin
      { GET CUR RECORD }
      CurRec.Init(Table.Table);
      CurRec:= Table.RecordGet(i);

      { EDIT RECORD }
      CurRec.SetFieldValue(NewField, TRUE);

      { WRITE BACK RECORD }
      if NOT Table.RecordUpdate(CurRec)                       <-------- Error here. It returns false
      then Log.AddError('Error adding record');             
     end;

    Table.UpdateToFile;
  FINALLY
    FreeAndNil(Table);                                                       
  END;
end;