mORMot Open Source

LB

Member

Registered: 2014-07-17

Posts: 2

BigTable with UInt64 key?

Hello Arnaud, All,

I'm looking for an alternative to sqlite3 for high performance insert/updates and BigTable seems a good candidate.
All the primary keys of my current schema are coded as unsigned Int64
Is there a way to use BigTable with UInt64 IDs? (I only saw integer or string IDs in the source code)

Thanks a lot,
Laurent

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 13,165

Website

Re: BigTable with UInt64 key?

We developed the unit on Delphi 7 as main platform, and there was no true UINt64 on this version yet...
This is the reason why we did stick at Int64 IDs, but stored as unsigned integers AFAIR.

I'm sad to say that our SynBigTable unit is more a Proof Of Concept than an active project.
We had some unfixed issues on production, and went back to SQLite3 or TSQLRestStorageInMemory kind of storage.

In BATCH mode + exclusive/off settings, SQLite3 is very fast.
More than 150,000 records per second, with our ORM layer, and direct SQlite3 access, with a transaction, and in BATCH mode.
See e.g. http://blog.synopse.info/post/2014/05/0 … -benchmark

LB

Member

Registered: 2014-07-17

Posts: 2

Re: BigTable with UInt64 key?

Thanks for your answer,
I currently use a sqlite3 wrapper, but not yours, which seems more efficient. Will give it a try.
Regards,
Laurent

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 13,165

Website

Re: BigTable with UInt64 key?

For best performance with SQLite3, ensure that:
- you use PRAGMA locking_mode = EXCLUSIVE
- you use PRAGMA synchronous = 0
- you nest all your writes within a transaction
- you re-use a prepared statement with bound parameters
- your statement is a multiple insert
- test if WAL mode and Memory Mapped IO is interresting in your case

sh17

Member

Registered: 2011-08-09

Posts: 25

Re: BigTable with UInt64 key?

This means BigTable is dead? I use that very successfully.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 13,165

Website

Re: BigTable with UInt64 key?

We do not use it here any more...

But we will include any patch to the source.

sh17

Member

Registered: 2011-08-09

Posts: 25

Re: BigTable with UInt64 key?

<OT>Can I forking it at the appropriate time on github?</OT>

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 13,165

Website

Re: BigTable with UInt64 key?

Yes any input un welcome!