#2 Big Table » Need help to figure out if Big Table is good for me » 2014-04-03 16:21:02

newfedra
Replies: 4

Hi

I have some really big files. Usually they stay in 1-6GB range. But sometimes these file can be even up to 14GB large.
Each file has several millions of entries like this (this is a simplified example):

string_about_100_to_1000_chars   a_short_string   number1    number2     number3 ... number10                  <--------- there is no more than 2KB of data on each entry

I need to:
1. display all rows in a TStringGrid (this is easy because I can only load from disk the data associated with the visible rows)
2. to sort the data (when the user click a column header).

My questions are:

  1. can I use Big Table for this
  2. if yes, can you please provide a skeleton example and some basic concepts on how to start (to store my data into the table for 'fast' sorting on computers with 2-4GB RAM)?
  2. will it work also on Lazarus?

Many thanks

#3 Re: Delphi » I'm still preferring Delphi 7 IDE » 2011-07-13 17:30:48

I agree with Ab.
Delphi XE is really slow and buggy. It will take some time until they will release Update 2, 3, 4 (and maybe 5) to fix all those bugs.

#4 Big Table » How to subscribe to a new list? » 2011-02-22 10:18:50

newfedra
Replies: 1

Hi.
There is some way (email, forum, twitter, newslist, etc) to find out when a new version of BigTable (or other library) is released?

#5 Re: Source Code repository » Synopse Big Table 1.12b - major bug fix » 2011-02-22 10:15:23

>when all AddField() methods have been called

I imagined that but I had to be sure smile
Thanks.

#6 Re: Source Code repository » Synopse Big Table 1.12b - major bug fix » 2011-02-21 20:55:54

If I add more fields, I have to call Table.AddFieldUpdate after EACH AddField? Or just once, after all AddField calls is ok?

#8 Big Table » About tfoIndex/tfoUnique » 2011-02-15 16:27:14

newfedra
Replies: 1

Hi.
I have a question: When to use tfoIndex, tfoUnique, tfoCaseInsensitive?

Let's say I want to store in the table two strings, some binary data and some integers. Something like this:

fdName := Table.AddField('Name' , tftWinAnsi, [tfoIndex] );
fdBase  := Table.AddField('Base'  , tftWinAnsi, [] );
fdNoOf  := Table.AddField('NoOf'  , tftInt32  , [] );

When I read the table back, I need only few of its columns (the entire column) - I just read the columns in memory and process the data. So I don't do searches in the table. Is it ok is none of the fields have the tfoIndex and tfoUnique attribute set?

Also, if I set to tfoCaseInsensitive attribute, it will work faster (usually CaseInsensitive strings are faster, right)?
Thanks

#9 Re: Big Table » How to check if field exists? » 2011-02-15 16:15:03

> Some more simple sample is needed...
Yes. It took me quite a while until a figure out basic stuff (how to create/add/update records, etc).
smile

Anyway, now that I figured out most of these basic stuffs, the library is great.
I will continue to test it. If I find other strange stuff I let you know.

#10 Re: Big Table » How to check if field exists? » 2011-02-15 13:01:24

Thanks.

There might be another issue. This time in the example provided by you in your blog. The record obtained using the variant code is not 1 but 0. The ID obtained with the non-variant code is 1. Maybe I have done something wrong. Also maybe there is another bug. Just take a look.


...
    Table.UpdateToFile;

    { VERIFY - READ RECORD: }
    ReadRec:= Table.RecordGet(aRecID);
    assert(ReadRec.ID= aRecID);

    { a) CLASSIC }
    s:= ReadRec.GetFieldValue(fdBases);
    Form1.lblString.caption:= s;
    Assert(s= ctTestStr);

    { b) VARINAT }
    Vari:= Table.VariantGet(aRecID);
    vID:= vari.ID;
    assert(vari.Base= ctTestStr);
    Assert(vID= aRecID);                              // <-- this fails
...


------------------------

I have also a suggestion.
The example (TestBigTable) provided by you is EXTREMELY difficult to follow because it is a function that contina lots of other functions. It will be nice if the example will be provided in a separated file and without that function-in-function structure.
smile

#11 Re: Big Table » How to check if field exists? » 2011-02-15 11:02:50

Thanks.
Where (which page) will be the update released?

#12 Big Table » How to check if field exists? » 2011-02-14 20:18:41

newfedra
Replies: 10

Hi.
I want to create a table and and to write to it records. Each record contains a string and an integer:

  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1');       
  TRY
    fdBases   := Table.AddField ('Base'    , tftWinAnsi, [tfoIndex] ); 
    fdNoOfBas := Table.AddField( 'NoOfBase', tftInt32  , [tfoIndex, tfoUnique] ); 
    Table.AddFieldUpdate;     





Tomorrow I may add a boolean field also:

  procedure TForm1.WriteExtraFields;
  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1'); 
  TRY
    Table.AddField ('bool', tftBoolean);
    Table.AddFieldUpdate;       





But forget tomorrow. I want to read the table today, while the third field was not yet added. But the program crashes here:

      { Retrieve fields }
      aStr:= ReadRec.GetFieldValue(fdBases);
      aInt:= ReadRec.GetFieldValue(fdNoOfBas);
      bol := ReadRec.GetFieldValue(fdBoolean);                <-------- Crash here.


How do I test if a field exists before reading it. There is something like: "if Table.FieldExists then DoStuff" ?







----------------------
EDIT:


I also have problems updating the current records when adding the new column:

procedure TForm1.WriteExtraFields;
VAR
   i: Integer;
   Table: TSynBigTableRecord;
   CurRec: TSynTableData;
   NewField: TSynTableFieldProperties;
begin
  { Create table }
  Table:= TSynBigTableRecord.Create('MyFirstDB.DB','Tabel1');
  TRY
    { Define an extra fields/column to be added to the table }
    NewField:= Table.AddField ('bool', tftBoolean);
    Table.AddFieldUpdate;

    for i:= 1 to 10 DO
     begin
      { GET CUR RECORD }
      CurRec.Init(Table.Table);
      CurRec:= Table.RecordGet(i);

      { EDIT RECORD }
      CurRec.SetFieldValue(NewField, TRUE);

      { WRITE BACK RECORD }
      if NOT Table.RecordUpdate(CurRec)                       <-------- Error here. It returns false
      then Log.AddError('Error adding record');             
     end;

    Table.UpdateToFile;
  FINALLY
    FreeAndNil(Table);                                                       
  END;
end;

#13 Fast JPEG decoder » Is your JPEG library usable? » 2010-08-31 20:41:50

newfedra
Replies: 1

Hi.
I discovered recently your web site. Lots of nice articles!!

I have also found the fast JPEG decoder and intended to use it immediately, but lucky me I have found a post on this forum where you say that it won't with progressive JPEG files. Well, 90% of existing jpg files are progressive (which is the only way they should be!) so basically your library cannot be used in the real world. Or there is a trick?

Board footer

Powered by FluxBB