#151 mORMot 1 » litestream - A SQLite replication tool that uses WAL and checkpoints » 2021-01-30 06:13:57

edwinsn
Replies: 4

https://github.com/benbjohnson/litestream
Unfortunately, it has no Windows support yet, but it seems to be on the author's plan.

The way it works looks like a good idea to implement "sqlite replication":

How it works
===
SQLite provides a WAL (write-ahead log) journaling mode which writes pages to a -wal file before eventually being copied over to the original database file. This copying process is known as checkpointing. The WAL file works as a circular buffer so when the WAL reaches a certain size then it restarts from the beginning.

Litestream works by taking over the checkpointing process and controlling when it is restarted to ensure that it copies every new page. Checkpointing is only allowed when there are no read transactions so Litestream maintains a long-running read transaction against each database until it is ready to checkpoint.

The SQLite WAL file is copied to a separate location called the shadow WAL which ensures that it will not be overwritten by SQLite. This shadow WAL acts as a temporary buffer so that replicas can replicate to their destination (e.g. another file path or to S3). The shadow WAL files are removed once they have been fully replicated. You can find the shadow directory as a hidden directory next to your database file. If you database file is named /var/lib/my.db then the shadow directory will be /var/lib/.my.db-litestream.

Litestream groups a snapshot and all subsequent WAL changes into "generations". A generation is started on initial replication of a database and a new generation will be started if litestream detects that the WAL replication is no longer contiguous. This can occur if the litestream process is stopped and another process is allowed to checkpoint the WAL.

#152 Re: mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-30 03:47:53

@Junior,
Sorry to hear that and I'm glad you are recovering!

Take care, @ab and all.

#153 Re: mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-28 11:07:31

tbo wrote:
edwinsn wrote:

Maybe we can use mORMot sample 09 + Apace ab to do a side-by-side benchmark?
Maybe we can even ask all member on this forum to share about their VPS statistics? wink

Windows Server 2019
CPU: 4 vCore
RAM: 6 GB (min), 12 GB (max)
Storage: 200 GB SSD-Boost
Bandwidth: 1000 MBit / Traffic Flatrate
Price: 15 EUR/Month regular price / I pay 10 EUR/Month (promotional offer that exists from time to time)
webtropia

Hey Thomas, the webtropia specs looks better than vultr! Not sure about the stability and support...

Where did you get the offer and where I can possibly receive it wink

#154 Re: mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-28 04:08:27

Junior/RO wrote:
edwinsn wrote:

Yes, I do. Just checked, I pay $26/month for a Vultr Windows VPS, maybe $10 is the Windows license fee.
I also use linux VPS (AWS/digitalocean/Linode)

Actually, vultr is the only big-names that provide decent Windows VPS at a reasonable price - the others is either expensive or slow.

Do you know Contabo? I'm using for a year, now. Windows Serer 2016, 16 GB. 18 euros.
Vultr is better?

I don't know Contabo. When I was switching from Linux to Windows VPS, vultr is the only acceptable option to me. I first heard about it through good words on Hacker News - which is a trust worthy community to me.

Here is the stats:
CPU: 1 vCore
RAM: 2048 MB
Storage: 55 GB SSD
Bandwidth: 2000 GB/Month
Price: $26 (VPS: $10; Windows License Fee: $16 ) = eur21.5

Maybe we can use mORMot sample 09 + Apace ab to do a side-by-side benchmark?
Maybe we can even ask all member on this forum to share about their VPS statistics? wink

#155 Re: mORMot 1 » mORMot 2 in Good Shape » 2021-01-25 15:30:34

ab wrote:

#define SQLITE_ENABLE_SESSION
// enable Sessions https://sqlite.org/sessionintro.html
// - not used/published yet in pascal code

The session extension seems to be a perfect tool for implementing a one-way database sync across nodes!

#156 Re: mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-24 13:57:03

@pvn0, how old are you then? big_smile
I have the impression that vultr is quite new - it appeared after Digital Ocean IIRC...

#157 Re: mORMot 1 » Case-insensitive search Cyrillic in SQLite3 » 2021-01-24 06:07:16

Wow! Amazing work as usual. How can you implement it so quickly!

Although it's a pity no mORMot 1 support...

#158 Re: mORMot 1 » Case-insensitive search Cyrillic in SQLite3 » 2021-01-23 13:13:41

ab wrote:

This is why a hard-coded pure mORMot Unicode comparison could make sense.
It would be cross-platform and consistent over all systems, as SQlite3 should.
I will include it into mormot.core.unicode.pas from mORMot 2.

Yes! When the db is used as a document file format, the user might copy the file across different locals.
And yes, that hard-coded collation must be **unicode-ready**.
Thanks for your effort!

#159 Re: mORMot 1 » Case-insensitive search Cyrillic in SQLite3 » 2021-01-23 04:55:28

ab wrote:

So if you move a DB from one system to another, depending on the local, you may break the indexes...

@ab, what's the workaround to handle it after moving across locals? Delete and rebuild the indexes?

#160 Re: mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-22 09:41:41

@pvn0,

Yes, I do. Just checked, I pay $26/month for a Vultr Windows VPS, maybe $10 is the Windows license fee.
I also use linux VPS (AWS/digitalocean/Linode)

Actually, vultr is the only big-names that provide decent Windows VPS at a reasonable price - the others is either expensive or slow.

If you or everybody is interested, use my reference link, I think both you and me will get some deposit  wink

#162 mORMot 1 » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-01-21 05:13:45

edwinsn
Replies: 30

https://github.com/facebookarchive/grace

It's for golang, and not sure how's achieved, but its github page says it can upgrade your web server without dropping any connections.

#164 Re: mORMot 1 » SynCommons.SynVarDispProc interfering MS Word automation? » 2021-01-08 10:46:52

@ab,
That's Great! But since the comments (posted below) didn't mention any Delphi versions and the code is either `{$ifdef  DELPHI6OROLDER}` or `{$ifdef CPU64}`, so I actually couldn't read the info you provided above...

But thank you all the same!

{$ifdef DOPATCHTRTL}
  {$define DOPATCHDISPINVOKE} // much faster late-binding process for our types
{$endif}
{$ifdef CPU64}
  {$define DOPATCHDISPINVOKE}
  // we NEED our patched DispInvoke to circumvent some Delphi bugs on Win64
{$endif}
{$ifdef DELPHI6OROLDER}
  {$define DOPATCHDISPINVOKE}
  // to circumvent LIdent := Uppercase() in TInvokeableVariantType.DispInvoke()
{$endif}

{$endif FPC}

#165 Re: mORMot 1 » SynCommons.SynVarDispProc interfering MS Word automation? » 2021-01-08 10:26:32

I'm in trouble then... I use TDocVariant with late binding...

#166 Re: mORMot 1 » SynCommons.SynVarDispProc interfering MS Word automation? » 2021-01-08 05:46:35

ab wrote:

Try to disable DOPATCHDISPINVOKE conditional around line 46585.

I will try, but I found two lines of comments about this conditional define:
  // we NEED our patched DispInvoke to circumvent some Delphi bugs on Win64
  // to circumvent LIdent := Uppercase() in TInvokeableVariantType.DispInvoke()

Are these two items still true?

#167 mORMot 1 » SynCommons.SynVarDispProc interfering MS Word automation? » 2021-01-07 19:05:23

edwinsn
Replies: 6

Hello Arnaud,

My program uses Word automation late-binding, and today I found some EOleException errors are related to`SynCommons.SynVarDispProc`, in the following line:

      // process Ole Automation variants
      if Assigned(VarDispProc) then
        VarDispProc(pointer(Result),Variant(Instance),CallDesc,@Params); // line 46521

It's weird, it's not clear to me why madExcept's reports error in`SynCommons.SynVarDispProc` when calling a Word document object.

Do you think if the OleVariant interception by SynCommons will cause any EOleException with message like "command failed" and so on? How to avoid SynCommons.SynVarDispProc to be called?

Thanks.

Call stack:

00696a6f +0bf app1.exe System.Win.ComObj                    DispatchInvokeError
006963e1 +1f1 app1.exe System.Win.ComObj                    DispatchInvoke
00696992 +0b2 app1.exe System.Win.ComObj                    VarDispInvoke
008cc0b3 +0d3 app1.exe SynCommons                46521  +10 SynVarDispProc
017e8517 +167 app1.exe MyUnit1                     165  +35 MyClass1.Method1 // here the program calls the OleVariant

#168 Re: mORMot 1 » mORMot 2 proposal: rename RawUTF8 type to Utf8 ? » 2021-01-05 16:09:47

Good choices! Thanks ab for all your efforts!

#169 mORMot 1 » mORMot2 - possible to add custom SQL to the queue of TRestBatch? » 2021-01-02 15:18:34

edwinsn
Replies: 0

Hello ab,

Since TRestBatch's db operations are **limited to**  add/update/delete operations, is it possible to also execute custom SQL statements with TRestBatch?

That'll be amazing if it's possible with mORMot2!

#170 Re: mORMot 1 » mORMot 2 proposal: rename RawUTF8 type to Utf8 ? » 2021-01-01 07:20:45

tbo wrote:

But for proper names like SQLite, you should keep the spelling.
TSQLite3HttpServer looks better than TSqlite3HttpServer to me.

Sorry, but I have to disagree. Why the exception, it doesn't make sense. `TSqlite3HttpServer ` looks good and it's comply with the new naming convention. Maybe it takes a little to get use to it wink

And happy new year!

#171 Re: mORMot 1 » mORMot 2 proposal: rename RawUTF8 type to Utf8 ? » 2020-12-31 11:12:00

Eugene Ilyin wrote:

Maybe Kotlin (successor of all Android dev) balance is the best approach?

When using an acronym as part of a declaration name, capitalize it if it consists of two letters (IOStream); capitalize only the first letter if it is longer (XmlFormatter, HttpInputStream).

Hey, that's exactly the rule I came up with myself, and has been using it for a while wink

#172 Re: mORMot 1 » mORMot 2 proposal: rename RawUTF8 type to Utf8 ? » 2020-12-31 05:45:09

Eugene Ilyin wrote:

Also it will be not good at all to introduce another mORMot.UTF8String/System.UTF8String name collision with standard type.
Is it possible to mess it up when you use both: some Delphi libraries expected System.UTF8String and some mORMot UTF8String routines in the same code.

I highly agree with Eugene, especially the drawback of conflict with System.UTF8String in some cases!

#173 Re: mORMot 1 » mORMot 2 proposal: rename RawUTF8 type to Utf8 ? » 2020-12-30 16:00:42

I suggest to keep the `T` prefix.

`RawUTF8` is OK, the length of the name is reasonable, it's not lengthy at all. On the other hand, `UTF8` is too short in my eye...

#174 mORMot 1 » SQLite as a document database » 2020-12-27 07:25:06

edwinsn
Replies: 2

I'd like to share with you a very useful article about using the JSON and 'generated columns' features of SQLite for storing scheme-less documents:
https://dgl.cx/2020/06/sqlite-json-support

SQLite is sssssssoooooooooo powerful! I'm glad that @ab had the envision on it and have chosen it as the foundation as mORMot wink

#175 Re: mORMot 1 » Blobs and BatchUpdater » 2020-12-27 04:58:39

leus wrote:
ab wrote:

You can specify the fields.

But note that the Blob content will be serialized as JSON during transmission so it is only good for small Blob size .

This is interesting.

I need to do this operation once a day, and we are talking between 1,000 and 10,000 records with a blob size that may vary between 2kb up to 100kb (I don't have the number yet). Assuming worst case scenario, we are talking about 1GB in raw binary before any JSON transformation. Is it better to transfer one-by-one in this case?

Sending 1GB at a time is obviously not a good idea. You can check `TSQLRestBatch.SizeBytes` and when it reaches a certain number, for example, 5MB, then you send it, and start another batch for the remaining data.

#177 Re: mORMot 1 » Blobs and BatchUpdater » 2020-12-25 05:57:47

Yes, use ALL_FIELDS or your custom field bits for the `Add` and `Update` methods

#178 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-25 05:50:25

@ab,

Thanks! Since  now sqlite.org officially provides both win32 and win64 versions of DLL download, why not just use their version...Wait! I remember now, mORMot includes an encryption layer for sqlite...

So in other words, if I don't use the mORMot encryption layer for SQLite, I can just use the official dll from sqlite.org, right?

#179 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-24 15:49:26

@ab,

So I'll use the sqlite3.dll downloaded from https://www.sqlite.org/download.html for the Win64 target. I assume there will be no other drawbacks? Thanks!

#180 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-24 15:46:55

@zed,

Wow, thanks for the help and I appreciate that!

So I confirm sqlite3 dll from https://synopse.info/files/SQLite3-64.7z does NOT working with fts5 Trigram, but dll from https://www.sqlite.org/download.html works!

#181 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-24 06:25:56

@zed, correction: For win32 I use I use SynSQLite3Static.pas, for Win64 I use the sqlite3.dll

#183 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-22 06:34:16

@ab,

Would you advise a workaround for using hte new Trigramidx  FTS5 tokenizer? Thanks!

#185 Re: mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-21 10:30:44

OK, I figured out I can define a `TSQLRecordFTS5Trigram= class(TSQLRecordFTS5)' class to use the new tokenizer, and `TSQLRecord.GetSQLCreate` can generate the correct SQL.

But then I got the "no such module: fts5, extended_errcode=1" error...

#186 mORMot 1 » No support for trigram FTS5 tokenizer added in SQLit3.34.0(2020-12-01) » 2020-12-21 10:19:17

edwinsn
Replies: 13

@ab,

SQLite Release 3.34.0 On 2020-12-01 added a so-called trigram tokenizer for FTS5, and it's not integrated into `TSQLRecord.GetSQLCreate` yet in mORMot 1.

Would you add support for that? Thanks!

#187 Re: mORMot 1 » The Trim in SysUtils conflicts with the Trim defined in SynCommons » 2020-11-16 15:36:02

ab wrote:

SpringCloud is indeed interresting.

Currently, with mORMot 1.18 there are the basic bricks of daemons setup and administration - and they are used in production - but they need to be glued by some proprietary code.

With mORMot 2, we would like to offer something similar. We will integrate the daemons and their hosting, with a centralized administration tool. Including remote real-time logging, and remote database access - which are features not available in SpringCloud.

We will focus on MicroServices definition and hosting.
We will define services from a set of settings files, as part of the source code, to setup the daemon name/ports/persistence features.
Then hosting would be as simple as assigning daemons to hosts. Hosts will be first regular Windows and Debian/Ubuntu Linux instances (since we know them very much) but could later on be some Cloud providers (even if the fully-embedded nature of mORmot MicroServices doesn't make cloud services useful). A nginx front-end (with Let'sEncrypt TLS) would be automatically setup on each Linux host - or as stand-alone frontend to Windows hosts. Hosts will be monitored in real-time (processes, memory, network). Also executable updates will be done remotely.
Security will be offered using asymetric public/private keys, via mormot.core.ecc.pas.
We did not start to work on it yet. But it is something we will implement, since we need it for the future.

Sounds very cool!

#188 mORMot 1 » [Suggestion] Support '\n', '\t' in ForumatUtf8 » 2020-11-11 12:03:18

edwinsn
Replies: 1

@ab,

It's very common to use '\n' to represent a 'new line', and '\t' to represent a TAB character in the programming.
In the Delphi community, I know at least DK Language supports these notions.
It'll be a nice addition if we can add those support in FormatUtf8 wink

#189 Re: mORMot 1 » Need general advises for a lot of small databases & cached records. » 2020-11-09 10:38:25

Great! Thanks! Excellent support of an open source project, as always!

#190 Re: mORMot 1 » Need general advises for a lot of small databases & cached records. » 2020-11-09 08:54:46

@ab,

Thanks! Very helpful!

I also noticed TSQLDataBase.SetUseCache  will create a TSynCache object:

TSynCache.Create(16384*1024,true)

which is a 16-Mb framework-level cache.

Unfortunately, the size of the TSynCache size is hard-coded. Any chance to make it configure-able? wink

#191 mORMot 1 » Need general advises for a lot of small databases & cached records. » 2020-11-08 18:03:52

edwinsn
Replies: 4

I have a project might be a little different than the usual use cases, and I want some general advises as to reducing **memory use** and **improving performance**.

- The program will open a folder which might contain a lot of files and subfolders.
- In each subfolder there is a sqlite db and its corresponding TSQLRestServerDB be created.
- for each file the program will create an object of a TSQLRecord-decendent for it.

The problem is that for folders with many subfolders, the RAM consumption go up.

Any options to reduce the memory consumption for TSQLRestServerDB  and TSQLRecord objects?

Thanks!

#192 Re: mORMot 1 » mORMot 2.0 status? » 2020-11-08 16:39:18

@ab,
Thanks for the details. And "no compile warnings" is really an extraordinary achievement - even the Delphi RTL will produce a lot of warnings IIRC wink

#193 Re: mORMot 1 » mORMot 2.0 status? » 2020-11-07 15:50:13

I'm impressed by ab's slides on the mORMot2 status and I really like what I've seen so far!

I like the modular design. mORMot2 can be a very very practical foundation for writing cross-platform and cross-compiler software!

One question, I noticed an roadmap item about remote administration UI, will it be a desktop UI or web ui?

#194 Re: mORMot 1 » Infinity and beyond » 2020-11-06 15:54:55

@pvn0, agreed, the scripting language has never been the issue for me to do web frontend, the lack of a RAD way of doing the UI is.

Nowadays I prefer htmx.org (successor of intercooler.js) to do the basic stuff and jquery does the rest.

#195 Re: mORMot 1 » mORMot 2.0 status? » 2020-11-06 06:31:19

I'll definitely check the presentations out!

#196 Re: mORMot 1 » SOA - inject extra data into created object » 2020-11-05 14:53:18

radexpol wrote:

No way, my customer does not have information about database connection lol

What I meant is the client passes in a 'ClientId', which allows your service method to get/initialize the actual connection string.

But yes, ab had the perfect answer already wink

#197 Re: mORMot 1 » Renaming TSQLRecord to TORM » 2020-11-05 10:51:48

Good decision. Just as I thought wink

#198 Re: mORMot 1 » SOA - inject extra data into created object » 2020-11-05 03:48:46

I guess you should/can pass it using a parameter of the GetQueues method?

#199 Re: mORMot 1 » Renaming TSQLRecord to TORM » 2020-11-05 03:44:50

No conclusion yet from @ab as to whether or not to use the universal 'TSyn' type prefix wink

#200 Re: mORMot 1 » Renaming TSQLRecord to TORM » 2020-10-30 13:57:05

EMartin wrote:

I agree with prefixing the framework with TSyn*, historically Delphi third party frameworks have used this. Even more, functions and procedure they should have a prefix, but thats is more complex for compatibility. It's just another opinion.

TSyn prefix for all types are good, but I don't think we need prefix for procedures and functions - it's not a common practice  IMHO. We can just use fully qualified names for procedures and functions. for example: `unit1.procedure1`

Board footer

Powered by FluxBB