#1 2014-04-03 16:21:02

newfedra
Member
Registered: 2010-08-31
Posts: 14

Need help to figure out if Big Table is good for me

Hi

I have some really big files. Usually they stay in 1-6GB range. But sometimes these file can be even up to 14GB large.
Each file has several millions of entries like this (this is a simplified example):

string_about_100_to_1000_chars   a_short_string   number1    number2     number3 ... number10                  <--------- there is no more than 2KB of data on each entry

I need to:
1. display all rows in a TStringGrid (this is easy because I can only load from disk the data associated with the visible rows)
2. to sort the data (when the user click a column header).

My questions are:

  1. can I use Big Table for this
  2. if yes, can you please provide a skeleton example and some basic concepts on how to start (to store my data into the table for 'fast' sorting on computers with 2-4GB RAM)?
  2. will it work also on Lazarus?

Many thanks

Last edited by newfedra (2014-04-03 16:38:27)

Offline

#2 2014-04-03 20:08:38

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 13,165
Website

Re: Need help to figure out if Big Table is good for me

Some ideas:

BigTable won't be so much optimized for it, due to the way it works.

I recommend using memory mapped files, and the technique similar to TSynLogFile as defined in SynCommons.pas.
But TSynLogFile relies on TMemoryMapText which maps the file with only one view, so is limited to 3GB.
So you can't use it directly.
But this kind of algorithm will give amazing speed.
For instance, opening and indexing a huge log (500 MB or more) with TSynLogFile is as fast as the drive reads it. smile

Thanks for the interest.
smile

Offline

#3 2014-04-03 21:49:26

newfedra
Member
Registered: 2010-08-31
Posts: 14

Re: Need help to figure out if Big Table is good for me

Ok. Many thanks.

Offline

#4 2014-04-03 22:52:31

lkessler
Member
Registered: 2011-10-17
Posts: 4

Re: Need help to figure out if Big Table is good for me

Is there any way to memory map files larger than 3 GB in 64 bit processing?

Offline

#5 2014-04-03 23:39:17

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 13,165
Website

Re: Need help to figure out if Big Table is good for me

Yes in win64 you can map at file size in one view.

Offline

Board footer

Powered by FluxBB