mORMot Open Source

the code is public now: https://github.com/flydev-fr/mormot2-qwen-tts

the reference you asked for is in tests/data/infer-ref/qwen3-tts-lm, in your infer-ref format. It has embeddings in and logits out for both Qwen3 stacks of the model, computed by PyTorch in float32, so you can drop the folder into mormot2-ai as is.

one more finding: the Vulkan backend caches weights by host address, so weights freed and reallocated at the same address come back stale.

I've done some testing, the goal was for a hobby project, to run a TTS model natively in delphi, no python at all. Model is Qwen3-TTS-12Hz-0.6B-CustomVoice. The setup:
delphi 13, mormot 2.4 trunk, Win64, win11 and an AMD RX 9060 XT gpu through Vulkan.

What I reused from mormot2-ai:
- safetensors loader, Qwen2 BPE tokenizer, CPU backend (AVX2 kernels + thread pool), sampler
- the Vulkan backend: Q8_0 linear shaders, plus rmsnorm, qkrope, attention, swiglu and addnorm, recorded the way TQwen3Vulkan.Chunk does it

What I wrote on top:
- the talker and the code predictor - Qwen3 stacks fed with embeddings instead of token ids
- the 12 Hz audio decoder (causal convolutions, SnakeBeta, sliding-window transformer)
- frame-by-frame streaming to waveOut

The 12 Hz decoder runs on the GPU as well, with four small compute shaders of my own (a tiled conv-as-GEMM with fused epilogues, SnakeBeta, depthwise conv, LayerNorm), built on the TInferVulkan batch API. It is x29 real time on its own, and its wav stays within 5e-6 of PyTorch.

Correctness, checked layer by layer against the official python package (qwen_tts, PyTorch, float32, greedy):
- the decoder wav matches within 2.5e-6
- in FP32 on CPU the generated codes are identical to PyTorch on 4 test sentences (french, english, accents, a 12s text)

Perf, same machine and same model:

 
                 python bridge (ROCm)   mormot2-ai, native
speed            x0.8-1.2 real time     x5.9 real time
first sound      ~2 s                   ~95 ms
model load       -                      ~1.4 s
VRAM             -                      ~0.9 GB (Q8_0 talker + F32 decoder)
on disk          ~9.5 GB of pytorch     nothing but the 2.4 GB model

Talker and code predictor run on Vulkan in Q8_0, about 12 ms per 80 ms audio frame. The CPU-only fallback (Q8_0) still runs the talker at x3.3 real time.

A few things I ran into, just in case it helps someone:
- InferRowToF32 doesn't read BF16, which is the format of the TTS weights - I widened the rows myself
- TBpeTokenizer only loads tokenizer.json. This model ships vocab.json + merges.txt, so I subclassed it and called the protected Build
- TQwen3Vulkan only accepts token ids. An entry point taking embeddings would make it reusable for TTS and multimodal models - for now I just copied the push-constant records to rebuild the same pass
- with the KV cache in F16 my checks drifted by ~2.3e-2 relative, against 2.6e-5 in F32. I keep F32, the context is short anyway
- a TInferThreadPool work item that allocates on the heap gets serialized by the memory manager - a 22-thread Q8 conversion took 3.5s instead of 0.6s, a stack buffer fixed it

great work @sh17

edit: a few precisions, the 12 Hz decoder is not entirely on the GPU: the quantizer, the pre-conv and the sliding-window transformer stay on the CPU, only the heavy part (upsampling, ConvNeXt and the convolution blocks) runs on the GPU, and
- decoder wav vs PyTorch: within 2.5e-6 on the CPU, 4.8e-6 on the GPU

the marmot is alive - it's my automation, not the data.

github has been firing the scheduled workflow 6-8h late since oct 10 to actual ~10-11 UTC. My review step runs at 06:00 and gives up at 06:49, so the payload lands after its own window and then gets skipped the next morning. I wasn't aware that github cron is best-effort.

Fixed it by letting the llm check more often instead of relying only on the cron. It's running and should be back later this evening

ps: And I will be back about my last PR you fixed with some others findings

I have to go away, I created a gist with the explanation, patch and logs if it helps: https://gist.github.com/flydev-fr/de9c2 … 0222bd54f9

Share!!

a quick note about a related project: PWeb (or mtron, name isn't decided yet) - github.com/flydev-fr/pweb

it is a desktop runtime for FPC/Lazarus (delphi soon), in the spirit of Electron or Tauri: mormot2 plus the OS WebView (WebView2, WebKitGTK, WKWebView), with a JS, React or pas2js frontend. There is no HTTP between the UI and Pascal. The UI calls regular mORMot 2 interface-based services in-process, through TRestServer.Uri(), with no socket and no port.

If a pas2js implementation would help your work, feel free to look at:

• sdk/pas2js - pweb.native.pas: a promise-based Service.Method call with named JSON arguments and typed errors, plus its test suite

• examples/05-pas2js - a complete pas2js frontend calling a mormot service

• tools/templates/pas2js - the project template

v := await(JSValue, PWebInvoke('CalculatorService.Add', New(['a', 20, 'b', 22])));

the transport there is the WebView bridge rather than HTTP, but the call shape is transport-agnostic. It might fit behind a Mustache-generated client.

About mobile: Android and iOS are planned. The first desktop release (Windows, Linux, macOS) should be out very soon, and the mobile targets come next (android_webview, JAVA layer are already set up).

The run/tests are green and the patch file is there: mormot2-2026-09-16-abi-fixes.patch

if everything make sense, I can send a PR if you want.

edit 2:  mormot2 regression suite is missing after patch test to really confirm, they should be ran on every platform, I will add them this afternoon.

thanks! good timing, I've just been asked to add report generation and pdf export on our backend, and I never touched the pdf side of mormot until now. will give it a try this week.

Understood, thanks.

yes, this tool is useful!

If any of you have ideas on how to set up the workflows for code review, I'll take them - my attempts fail so far, and it's a cost and logic problem.

"edition": "2026-09-03",
"totals": {
  "commits": 22,
  "sessions": 61,
  "cost_usd": 54.5882,
  "turns": 1198,
  "seconds": 11152
}

$54 and 1198 turns on claude-opus_&_sonnet-5, for one day of commits. 3 hours of model time I should probably go read how Mozilla runs their security tooling minus Mythos llm

PS: it runs on a personal subscription so those dollars are an API-list equivalent, not a bill. Just a way to compare runs.

Sorry, merged, thanks

I had scheduled another LLM on the same commits, and this morning I got this one: https://github.com/flydev-fr/mormot2-da … 5285473602

imo it's more relevant, especially on the mistake you pointed out - it got the `var variant` part right:
(the #3 - MEDIUM: ORM Variant lifetime leak fixed, plus TOrmTable Variant-generation refactoring)

And it links to your blog post from yesterday. I'll let both run in parallel for a few days and compare before picking one but I think this last is far more specific.

Sure, you can grab it from github: x64callmethod.o
It's assembled from src/x64callmethod.asm with ml64, .debug$S stripped and timestamp zeroed; CI only publishes it after it passed the full suite in a linked executable.
You can also rebuild it with pwsh tools/make-dist.ps1.

Two notes about this one:
1. it contains both fixes: unwind metadata and imvCurrency RAX
2. it hard-codes TCallMethodArgs layout

I hit this exact FPC/Win64 issue yesterday with mormot2, on version 3.2.2 and 3.2.3. I found a workaround which seems to fix the exception unwinding reliably. I have also been able to reproduce and test a small MASM replacment quite extensively with regression test around it (one additional issue appeared during that testing about fpc returns imvCurrency through RAX).

If you are interested, I can post the details and I can clean this into a minimal standalone patch/test case on a github repo?

Just cross linking two threads here for reference:

https://synopse.info/forum/viewtopic.ph … 620#p41620 (mormot.i18n.pas)
https://synopse.info/forum/viewtopic.php?id=7337

I can't wait for i18n support to be added to

Done. PS: I have no idea what I’m doing when it comes to licensing

Hi all,

@sh17 sorry for the delay

Following up on my last post - the code goes public today: https://github.com/flydev-fr/mormot.ai

Two independent pieces under the mormot.ai.* namespace, both mormot2 only, no external dependencies:

  1. LiteLLM SDK port (mormot.ai.litellm.*): Its an unified LLM client - one call, any provider.
  2. pgvector / RAG stack: Vector search and RAG over pascal source code: TPgVector binary wire-format codec, a pascal source chunker, an embedding-client interface, hybrid retrieval blending cosine similarity and PostgreSQL tsvector, a RAG orchestrator, and three CLI programs, and  an attempt to ingest mormot2 code and build a chatbot using ollama/devstral-2, actually working.

There is a full docs site covering both parts: https://mormot-ai.fsb.dev
(serving the published "docs" folder).

Fair warning: the code works but is not finished, and the docs track it closely rather than exactly. Usable, not perfect.

Bonus: this documentation site is built and served with Mormotsaurus, a small CLI tool for authoring and publishing static documentation sites - itself written with mormot2 mvc layer. Windows and Linux binaries are in the repo for now; it will get its own repository soon.

Hello,

@sh17 I will push what I have.  Also, @ab is right, JP's neural-api is really great!

Thanks. Yes, the extensions are meant to be written using mormot only. The spec revision will be updated next week to the latest version.

Regarding authorization, it is indeed planned and will follow MCP security/authorization, with optional support for our secure RESTful authentication scheme.

I don’t know about @ab’s decision, but FYI, a few weeks ago I wrote a mormot-only port of LiteLLM to add an AI gateway providing simplified model access. It is still not polished, as I was missing the now-available streamable HTTP transport and will be pushed to the extensions repo soon.

Do not hesitate to reuse any code and to let use know about any update on your side

edit: small note,

This demo use the mormot.ext.os extension, more info there.

And here I am, missing yet another event.. This time it was actually geographically accessible, but I moved to the Balkans four days ago. meh

Insane - and absolutely deserved.

bravo!

Yes of course, and what you mean by "the service cannot work properly"?   Try to first accept un-safe cert from the browser, I mean, just open the url of the service api call on a browser-tab, accept un-safe connection once, then api calls will work.

// - secTLS will use HTTPS secure connection (user-provided self-signed or not)
// - secTLSSelfSigned will use HTTPS secure connection with a (temporary) self-signed certificate (will generate an temporary self-signed certificate at startup)

yes ok - confirmed on my side too.

it's happening each run if using a good workload. no exception, it happen always.

and useHttpAsync doesn't suffer from it.

i saved screenshots yesterday: