mORMot Open Source

tfopc

Member

Registered: 2024-01-08

Posts: 40

HTTP/2 support

Hi all,

does anybody know if it is possible to create a http/2 client that can send data to some endpoint?
In my case i need to fill a datawarehouse db (one table) with data from different endsystems (several GB per Client)

The data should be written to a duckdb.

At the moment i send and fill it initial per client via parquet - format, but i want to use a http endpoint.

Found ducktape (https://github.com/artie-labs/ducktape), that seems to performe great for several million records.
But the append endpoint need to be served via streams (NDJSON data) over HTTP/2

My research brings me to "DelphiGrpc" that uses ngHttp2 like curl does. Maybe it is an option to integrate ngHttp2 as option also in mormot httpclient?

And than - maybe "SynProtoRTSPHTTP.pas" is going in the right direction for an implementation at application level?

Thanks,
Tobias

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,347

Website

Re: HTTP/2 support

HTTP/2 is not supported yet, and there is no plan to implement it.
It is clearly complex, and has benefits only for concurrent process (e.g. loading images for a web page), which does not make sense for a client/server REST process.
And HTTP/3 aka QUIC is also there, and is more complex, but lighter on the network since it uses UDP.
We rather rely on our own REST encapsulation over WebSockets, which is more simple and fast enough, addressing most of HTTP/2 ideas (light headers, multiplexing, bidirectional).

Several GB per client is not much, and is easily handled with plain HTTPS.
There is no speed benefit of HTTP/2 over HTTPS with a single huge packet.
I don't see why you want to use ducktape, which is another third party service, just to gain a few percents for a single large query.
It may help only, and up to 30%, for small concurrent queries on a single client, which is not your usecase. But to fill a DB from a client, it is no silver bullet: the easiest is to prepare the DB on the client side, and HTTPS will saturate the TCP/TLS link as much as HTTP/2.
Even the DuckDB core itself was not design for high concurrency: https://duckdb.org/docs/stable/connect/ … oncurrency

tfopc

Member

Registered: 2024-01-08

Posts: 40

Re: HTTP/2 support

Hi ab,

thanks for you quick response.

Why ducktape? - it was the only service i found that give me the possibility to load data directly in the duckdb-table. At the moment i write from my win clients via http (mormot http-client) to our linux dashboard server in a postgres db via prestd (max 200 records per request). Works great so far. And the initial load is done via parquet - files. But my tests on the BI/Dashboard with postgres as datasource are not so good.

One big table with the sample-transaction data (15 Mio - records) with a join to customer-data
- Postres- PG18.1, Docker : 20sec for an analytical query
- DuckDB: 500ms for the same.

I will check to directly use duckdb on client side and write via postgres extension to a s3-bucket in DuckLake, or parquet - format.

https://duckdb.org/docs/stable/data/par … ned_writes

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,347

Website

Re: HTTP/2 support

I did not know much about DuckDB but it sounds like a nice C API.
In fact, I asked some AI about it, and it sounds like if it could be a very nice backend to be added to mORMot 2, in addition to its existing in-memory storage, Sqlite3, SQL and MongoDB.
https://gist.github.com/synopse/818d1e0 … ab24592bd0

We already included with success MongoDB as DB backend, able to leverage the best of ORM and SQLite3, using MongoDB mainly as centralized remote data storage from several microservices: each microservice has its own in-memory and SQLite3 storage, but efficiently push its data into the main shared database, mainly for archival purpose and analytics queries.
Since DuckDB is embedded and pure C, it could be a premium new storage engine for analytics, without the need to add another layer.
It won't be "another SQL" engine in addition to SQLite3 for instance. It would be an OLAP engine to be used in conjunction to the existing OLTP engines: DuckDB targets analytical, read-mostly, columnar workloads.
See https://github.com/synopse/mORMot2/issues/423

tfopc

Member

Registered: 2024-01-08

Posts: 40

Re: HTTP/2 support

Great!
I'm very exited to see what is coming in the future.

Maybe you realize aready that there are some Delphi projects on github that already speaks with DuckDB.

As far as i read the concept the AI is a fan of DuckDB and would love to see it integrated your Mormot

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,347

Website

Re: HTTP/2 support

Yes, but the idea is to go one step further and integrate it to our ORM and REST layer, mixing it with other SQL and NoSQL engines available in mORMot.

esmondb

Member

From: London

Registered: 2010-07-20

Posts: 300

Re: HTTP/2 support

Please keep it KISS