You are not logged in.
https://github.com/benbjohnson/litestream
Unfortunately, it has no Windows support yet, but it seems to be on the author's plan.
The way it works looks like a good idea to implement "sqlite replication":
How it works
===
SQLite provides a WAL (write-ahead log) journaling mode which writes pages to a -wal file before eventually being copied over to the original database file. This copying process is known as checkpointing. The WAL file works as a circular buffer so when the WAL reaches a certain size then it restarts from the beginning.
Litestream works by taking over the checkpointing process and controlling when it is restarted to ensure that it copies every new page. Checkpointing is only allowed when there are no read transactions so Litestream maintains a long-running read transaction against each database until it is ready to checkpoint.
The SQLite WAL file is copied to a separate location called the shadow WAL which ensures that it will not be overwritten by SQLite. This shadow WAL acts as a temporary buffer so that replicas can replicate to their destination (e.g. another file path or to S3). The shadow WAL files are removed once they have been fully replicated. You can find the shadow directory as a hidden directory next to your database file. If you database file is named /var/lib/my.db then the shadow directory will be /var/lib/.my.db-litestream.
Litestream groups a snapshot and all subsequent WAL changes into "generations". A generation is started on initial replication of a database and a new generation will be started if litestream detects that the WAL replication is no longer contiguous. This can occur if the litestream process is stopped and another process is allowed to checkpoint the WAL.
@Junior,
Sorry to hear that and I'm glad you are recovering!
Take care, @ab and all.
edwinsn wrote:Maybe we can use mORMot sample 09 + Apace ab to do a side-by-side benchmark?
Maybe we can even ask all member on this forum to share about their VPS statistics?Windows Server 2019
CPU: 4 vCore
RAM: 6 GB (min), 12 GB (max)
Storage: 200 GB SSD-Boost
Bandwidth: 1000 MBit / Traffic Flatrate
Price: 15 EUR/Month regular price / I pay 10 EUR/Month (promotional offer that exists from time to time)
webtropia
Hey Thomas, the webtropia specs looks better than vultr! Not sure about the stability and support...
Where did you get the offer and where I can possibly receive it ![]()
edwinsn wrote:Yes, I do. Just checked, I pay $26/month for a Vultr Windows VPS, maybe $10 is the Windows license fee.
I also use linux VPS (AWS/digitalocean/Linode)Actually, vultr is the only big-names that provide decent Windows VPS at a reasonable price - the others is either expensive or slow.
Do you know Contabo? I'm using for a year, now. Windows Serer 2016, 16 GB. 18 euros.
Vultr is better?
I don't know Contabo. When I was switching from Linux to Windows VPS, vultr is the only acceptable option to me. I first heard about it through good words on Hacker News - which is a trust worthy community to me.
Here is the stats:
CPU: 1 vCore
RAM: 2048 MB
Storage: 55 GB SSD
Bandwidth: 2000 GB/Month
Price: $26 (VPS: $10; Windows License Fee: $16 ) = eur21.5
Maybe we can use mORMot sample 09 + Apace ab to do a side-by-side benchmark?
Maybe we can even ask all member on this forum to share about their VPS statistics? ![]()
#define SQLITE_ENABLE_SESSION
// enable Sessions https://sqlite.org/sessionintro.html
// - not used/published yet in pascal code
The session extension seems to be a perfect tool for implementing a one-way database sync across nodes!
@pvn0, how old are you then? ![]()
I have the impression that vultr is quite new - it appeared after Digital Ocean IIRC...
Wow! Amazing work as usual. How can you implement it so quickly!
Although it's a pity no mORMot 1 support...
This is why a hard-coded pure mORMot Unicode comparison could make sense.
It would be cross-platform and consistent over all systems, as SQlite3 should.
I will include it into mormot.core.unicode.pas from mORMot 2.
Yes! When the db is used as a document file format, the user might copy the file across different locals.
And yes, that hard-coded collation must be **unicode-ready**.
Thanks for your effort!
So if you move a DB from one system to another, depending on the local, you may break the indexes...
@ab, what's the workaround to handle it after moving across locals? Delete and rebuild the indexes?
@pvn0,
Yes, I do. Just checked, I pay $26/month for a Vultr Windows VPS, maybe $10 is the Windows license fee.
I also use linux VPS (AWS/digitalocean/Linode)
Actually, vultr is the only big-names that provide decent Windows VPS at a reasonable price - the others is either expensive or slow.
If you or everybody is interested, use my reference link, I think both you and me will get some deposit ![]()
Currently I put it behind an IIS ![]()
https://github.com/facebookarchive/grace
It's for golang, and not sure how's achieved, but its github page says it can upgrade your web server without dropping any connections.
Great job as always!
@ab,
That's Great! But since the comments (posted below) didn't mention any Delphi versions and the code is either `{$ifdef DELPHI6OROLDER}` or `{$ifdef CPU64}`, so I actually couldn't read the info you provided above...
But thank you all the same!
{$ifdef DOPATCHTRTL}
{$define DOPATCHDISPINVOKE} // much faster late-binding process for our types
{$endif}
{$ifdef CPU64}
{$define DOPATCHDISPINVOKE}
// we NEED our patched DispInvoke to circumvent some Delphi bugs on Win64
{$endif}
{$ifdef DELPHI6OROLDER}
{$define DOPATCHDISPINVOKE}
// to circumvent LIdent := Uppercase() in TInvokeableVariantType.DispInvoke()
{$endif}
{$endif FPC}I'm in trouble then... I use TDocVariant with late binding...
Try to disable DOPATCHDISPINVOKE conditional around line 46585.
I will try, but I found two lines of comments about this conditional define:
// we NEED our patched DispInvoke to circumvent some Delphi bugs on Win64
// to circumvent LIdent := Uppercase() in TInvokeableVariantType.DispInvoke()
Are these two items still true?
Hello Arnaud,
My program uses Word automation late-binding, and today I found some EOleException errors are related to`SynCommons.SynVarDispProc`, in the following line:
// process Ole Automation variants
if Assigned(VarDispProc) then
VarDispProc(pointer(Result),Variant(Instance),CallDesc,@Params); // line 46521It's weird, it's not clear to me why madExcept's reports error in`SynCommons.SynVarDispProc` when calling a Word document object.
Do you think if the OleVariant interception by SynCommons will cause any EOleException with message like "command failed" and so on? How to avoid SynCommons.SynVarDispProc to be called?
Thanks.
Call stack:
00696a6f +0bf app1.exe System.Win.ComObj DispatchInvokeError
006963e1 +1f1 app1.exe System.Win.ComObj DispatchInvoke
00696992 +0b2 app1.exe System.Win.ComObj VarDispInvoke
008cc0b3 +0d3 app1.exe SynCommons 46521 +10 SynVarDispProc
017e8517 +167 app1.exe MyUnit1 165 +35 MyClass1.Method1 // here the program calls the OleVariantGood choices! Thanks ab for all your efforts!
Hello ab,
Since TRestBatch's db operations are **limited to** add/update/delete operations, is it possible to also execute custom SQL statements with TRestBatch?
That'll be amazing if it's possible with mORMot2!
But for proper names like SQLite, you should keep the spelling.
TSQLite3HttpServer looks better than TSqlite3HttpServer to me.
Sorry, but I have to disagree. Why the exception, it doesn't make sense. `TSqlite3HttpServer ` looks good and it's comply with the new naming convention. Maybe it takes a little to get use to it ![]()
And happy new year!
Maybe Kotlin (successor of all Android dev) balance is the best approach?
When using an acronym as part of a declaration name, capitalize it if it consists of two letters (IOStream); capitalize only the first letter if it is longer (XmlFormatter, HttpInputStream).
Hey, that's exactly the rule I came up with myself, and has been using it for a while ![]()
Also it will be not good at all to introduce another mORMot.UTF8String/System.UTF8String name collision with standard type.
Is it possible to mess it up when you use both: some Delphi libraries expected System.UTF8String and some mORMot UTF8String routines in the same code.
I highly agree with Eugene, especially the drawback of conflict with System.UTF8String in some cases!
I suggest to keep the `T` prefix.
`RawUTF8` is OK, the length of the name is reasonable, it's not lengthy at all. On the other hand, `UTF8` is too short in my eye...
I'd like to share with you a very useful article about using the JSON and 'generated columns' features of SQLite for storing scheme-less documents:
https://dgl.cx/2020/06/sqlite-json-support
SQLite is sssssssoooooooooo powerful! I'm glad that @ab had the envision on it and have chosen it as the foundation as mORMot ![]()
ab wrote:You can specify the fields.
But note that the Blob content will be serialized as JSON during transmission so it is only good for small Blob size .
This is interesting.
I need to do this operation once a day, and we are talking between 1,000 and 10,000 records with a blob size that may vary between 2kb up to 100kb (I don't have the number yet). Assuming worst case scenario, we are talking about 1GB in raw binary before any JSON transformation. Is it better to transfer one-by-one in this case?
Sending 1GB at a time is obviously not a good idea. You can check `TSQLRestBatch.SizeBytes` and when it reaches a certain number, for example, 5MB, then you send it, and start another batch for the remaining data.
@ab,
Thanks for the detailed explanation!
Yes, use ALL_FIELDS or your custom field bits for the `Add` and `Update` methods
@ab,
Thanks! Since now sqlite.org officially provides both win32 and win64 versions of DLL download, why not just use their version...Wait! I remember now, mORMot includes an encryption layer for sqlite...
So in other words, if I don't use the mORMot encryption layer for SQLite, I can just use the official dll from sqlite.org, right?
@ab,
So I'll use the sqlite3.dll downloaded from https://www.sqlite.org/download.html for the Win64 target. I assume there will be no other drawbacks? Thanks!
@zed,
Wow, thanks for the help and I appreciate that!
So I confirm sqlite3 dll from https://synopse.info/files/SQLite3-64.7z does NOT working with fts5 Trigram, but dll from https://www.sqlite.org/download.html works!
@zed, correction: For win32 I use I use SynSQLite3Static.pas, for Win64 I use the sqlite3.dll
@zed, I use SynSQLite3Static.pas
@ab,
Would you advise a workaround for using hte new Trigramidx FTS5 tokenizer? Thanks!
FTS5 Trigramidx info page: https://sqlite.org/fts5.html#trigramidx
OK, I figured out I can define a `TSQLRecordFTS5Trigram= class(TSQLRecordFTS5)' class to use the new tokenizer, and `TSQLRecord.GetSQLCreate` can generate the correct SQL.
But then I got the "no such module: fts5, extended_errcode=1" error...
@ab,
SQLite Release 3.34.0 On 2020-12-01 added a so-called trigram tokenizer for FTS5, and it's not integrated into `TSQLRecord.GetSQLCreate` yet in mORMot 1.
Would you add support for that? Thanks!
SpringCloud is indeed interresting.
Currently, with mORMot 1.18 there are the basic bricks of daemons setup and administration - and they are used in production - but they need to be glued by some proprietary code.
With mORMot 2, we would like to offer something similar. We will integrate the daemons and their hosting, with a centralized administration tool. Including remote real-time logging, and remote database access - which are features not available in SpringCloud.
We will focus on MicroServices definition and hosting.
We will define services from a set of settings files, as part of the source code, to setup the daemon name/ports/persistence features.
Then hosting would be as simple as assigning daemons to hosts. Hosts will be first regular Windows and Debian/Ubuntu Linux instances (since we know them very much) but could later on be some Cloud providers (even if the fully-embedded nature of mORmot MicroServices doesn't make cloud services useful). A nginx front-end (with Let'sEncrypt TLS) would be automatically setup on each Linux host - or as stand-alone frontend to Windows hosts. Hosts will be monitored in real-time (processes, memory, network). Also executable updates will be done remotely.
Security will be offered using asymetric public/private keys, via mormot.core.ecc.pas.
We did not start to work on it yet. But it is something we will implement, since we need it for the future.
Sounds very cool!
@ab,
It's very common to use '\n' to represent a 'new line', and '\t' to represent a TAB character in the programming.
In the Delphi community, I know at least DK Language supports these notions.
It'll be a nice addition if we can add those support in FormatUtf8 ![]()
Great! Thanks! Excellent support of an open source project, as always!
@ab,
Thanks! Very helpful!
I also noticed TSQLDataBase.SetUseCache will create a TSynCache object:
TSynCache.Create(16384*1024,true)which is a 16-Mb framework-level cache.
Unfortunately, the size of the TSynCache size is hard-coded. Any chance to make it configure-able? ![]()
I have a project might be a little different than the usual use cases, and I want some general advises as to reducing **memory use** and **improving performance**.
- The program will open a folder which might contain a lot of files and subfolders.
- In each subfolder there is a sqlite db and its corresponding TSQLRestServerDB be created.
- for each file the program will create an object of a TSQLRecord-decendent for it.
The problem is that for folders with many subfolders, the RAM consumption go up.
Any options to reduce the memory consumption for TSQLRestServerDB and TSQLRecord objects?
Thanks!
@ab,
Thanks for the details. And "no compile warnings" is really an extraordinary achievement - even the Delphi RTL will produce a lot of warnings IIRC ![]()
I'm impressed by ab's slides on the mORMot2 status and I really like what I've seen so far!
I like the modular design. mORMot2 can be a very very practical foundation for writing cross-platform and cross-compiler software!
One question, I noticed an roadmap item about remote administration UI, will it be a desktop UI or web ui?
@pvn0, agreed, the scripting language has never been the issue for me to do web frontend, the lack of a RAD way of doing the UI is.
Nowadays I prefer htmx.org (successor of intercooler.js) to do the basic stuff and jquery does the rest.
I'll definitely check the presentations out!
No way, my customer does not have information about database connection
What I meant is the client passes in a 'ClientId', which allows your service method to get/initialize the actual connection string.
But yes, ab had the perfect answer already ![]()
Good decision. Just as I thought ![]()
I guess you should/can pass it using a parameter of the GetQueues method?
No conclusion yet from @ab as to whether or not to use the universal 'TSyn' type prefix ![]()
I agree with prefixing the framework with TSyn*, historically Delphi third party frameworks have used this. Even more, functions and procedure they should have a prefix, but thats is more complex for compatibility. It's just another opinion.
TSyn prefix for all types are good, but I don't think we need prefix for procedures and functions - it's not a common practice IMHO. We can just use fully qualified names for procedures and functions. for example: `unit1.procedure1`