Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

sh17 · 2026-06-17 21:03:11

Hi Arnaud, hi all,

We run mORMot 2 in production (FPC on Linux, native, no Indy/Wine) as the backend
of a SaaS for the trades. We now need an **MCP server** so our backend can expose
tools/resources to AI agents (e.g. Claude Desktop and our own agent), and ideally
LLM client drivers later on.

Before writing much code I'd like to check interest and get your guidance, because
I think this belongs **in the mORMot world**, not as yet another standalone lib.

## The gap

As far as I can see, mORMot has no AI/LLM/MCP module today. Meanwhile mORMot
already ships everything such a module needs and would do it better than the
Delphi-only options out there (which lean on Indy and `System.JSON`/`System.Rtti`):

- `mormot.net.server` / `mormot.net.async` — HTTP server for the *Streamable HTTP*
MCP transport
- `mormot.net.ws.*` — WebSockets for streaming
- `mormot.core.json` / `mormot.core.variants` — JSON-RPC 2.0 envelopes with
`TDocVariant`
- `mormot.core.rtti` — generating JSON-Schema for tool input from typed records
- cross-compiler (FPC + Delphi), cross-platform, no extra dependencies

## What I'm proposing

A small, **clean-room** extension under a new `mormot.ai.*` namespace, built
against the official **MCP specification (revision 2025-11-25)** and JSON-RPC 2.0
— not derived from any third-party MCP code. Initial, transport-neutral layout:

```
mormot.ai.mcp.types JSON-RPC/MCP types + envelope build/parse
mormot.ai.mcp.server engine: tool/resource registry + JSON-RPC dispatch
(initialize, tools/list, tools/call) — JSON in/out
mormot.ai.mcp.transport.stdio stdin/stdout (local subprocess transport)
mormot.ai.mcp.transport.http Streamable HTTP on mormot.net.server
```

The engine takes JSON in and returns JSON out, so it is fully unit-testable
without a socket; transports are thin adapters. Tools implement a small
`IMcpTool` interface (`GetName`/`GetDescription`/`GetInputSchema`/`Execute`).

I already have the first unit (`mormot.ai.mcp.types`) and FPCUnit tests for the
JSON-RPC envelopes; the rest is staged behind it.

## Questions

1. **Interest & home** — would you welcome such an extension under the
`mormot.ai.*` namespace (eventual upstream into mORMot), or would you prefer
it lives as an external companion package? Either is fine for us; I'd just
like to build it the way that has the best chance of being useful to others.
2. **Namespace & conventions** — if upstream is on the table, is `mormot.ai.*`
the naming you'd want (e.g. `mormot.ai.mcp.server`)? Any coding-style or CLA
requirements I should follow from the start?
3. **Transport** — for Streamable HTTP, what's your recommended building block:
plain `THttpServer`/`THttpAsyncServer` with manual chunked writes, or is there
a pattern you'd point me at for long-lived streaming responses + an optional
WebSocket upgrade?
4. **JSON-RPC API shape** — for a framework-grade API, would you prefer the
envelopes/dispatch built on `TDocVariant` (what I have now) or on typed records
with `mormot.core.rtti` serialization? Happy to follow your taste here.

I'm willing to maintain this and contribute it back. Thanks for mORMot — it has
been a joy to build a native FPC backend on it.

Best regards,
Sven

flydev · 2026-06-18 10:08:53

Hey @sh17, did you checked my mcp extension there? I pushed a new update 4 days ago that add streamable-http transport support (up to 2025-03-26 version and will be very soon updated to the latest).

Cross-platform. Fully testable. You can find also some demos.

code: mormot2-extensions/mormot-mcp-server

Last edited by flydev (2026-06-18 10:10:25)

sh17 · 2026-06-18 11:00:30

Hi flydev,

thanks a lot — I didn't know about mormot2-extensions, and your
mormot-mcp-server is genuinely the closest thing to "MCP done the mORMot way"
I've seen. I went through it in detail;

- It reads like a mORMot core unit, not a Delphi port: IInvokable tools,
RTTI-based schema generation, TDocVariant envelopes, the tri-license header
and * section style.
- The transport layer is excellent — stdio, HTTP, SSE and a real Streamable
HTTP with token-by-token streaming (incl. the Claude CLI demo). That directly
answers the streaming question from my post.
- The TSynTestCase coverage for the transports (origin validation, sessions,
batching, notifications, bad-JSON, 405 paths) is exactly the rigor I'd want.

Where my use case still diverges — and why I think there's room for both:

1. Spec revision. Our agents target the current MCP revision
(2025-11-25); your server is on 2024-11-05 + Streamable HTTP 2025-03-26.
For us the newer revision isn't optional.
2. Auth is our product core. We're a SaaS backend with JWT, HttpOnly refresh
cookies, __Host-csrf double-submit, a role/permission model and owner-capped
scopes. MCP tool execution has to run *inside* that context on our existing
mORMot host (TUriRouter + guards), not as a separate server. Your
TMcpAuthContext is a nice foundation, but the wiring is fairly specific to us.

So I'll keep building our mormot.ai.* clean-room layer (current spec + auth bound
to our context), but I'd love to align the Streamable HTTP transport with your
approach rather than reinvent it.

Which loops back to my question for Arnaud: with community MCP servers now existing,
is there appetite to converge any of this under an official namespace, or do you
prefer it stays as companion packages? Happy either way.

Thanks again for pointing me to it.

Best,
Sven

Last edited by sh17 (2026-06-18 11:02:41)

flydev · 2026-06-18 11:51:46

Thanks. Yes, the extensions are meant to be written using mormot only. The spec revision will be updated next week to the latest version.

Regarding authorization, it is indeed planned and will follow MCP security/authorization, with optional support for our secure RESTful authentication scheme.

I don’t know about @ab’s decision, but FYI, a few weeks ago I wrote a mormot-only port of LiteLLM to add an AI gateway providing simplified model access. It is still not polished, as I was missing the now-available streamable HTTP transport and will be pushed to the extensions repo soon.

Do not hesitate to reuse any code and to let use know about any update on your side

edit: small note,

sh17 wrote:

(incl. the Claude CLI demo). That directly answers the streaming question from my post.

This demo use the mormot.ext.os extension, more info there.

Last edited by flydev (2026-06-18 12:04:37)

ab · 2026-06-18 18:04:22

This is a very nice idea.

Perhaps it would be to include this whole mormot.ai.* part to the framework, because I didn't write it and can't test it really, so I can't maintain it.
But the idea of third-party/custom mormot units like mormot.ext.* is great.

BTW https://github.com/flydev-fr/mormot2-ex … /mormot-os is likely to finish in mormot, because it is really a missing basic feature.
Perhaps together with fixing https://github.com/synopse/mORMot2/issues/463 issue.

sh17 · 2026-06-18 18:42:52

If it’s okay with flydev, I’d like to propose some code that builds upon his—specifically, a `mormot.ai` namespace. I’m also willing to maintain it. I’ve implemented the current MCP specification, and right now, I’m working on LLM clients and agents.

Flydev's mormot-os is definitely required. The TLimitedStreamWriter you added today is already being put to use as well.

Sven

sh17 · 2026-06-18 18:49:18

By the way, I also developed a sqlite-vec extension using lembed, available at https://github.com/landrix/sqlite-vec-for-Delphi. It builds upon two extensions.
I'm exclusively using FreePascal now—no more Delphi. It's just because of the repo names.

Last edited by sh17 (2026-06-18 18:52:56)

flydev · 2026-06-18 19:15:27

That works for me - and I'll check what you did with the sqlite-vec extension, since I'm very interested in it and for sure will complete the db vector adaptors nicely (see below).

Back in May, while working on my LiteLLM port, I also wrote some code around LLMs, RAG, and similar topics, but targeting PostgreSQL through custom bindings for pgvector/pgvector - there are only abstract classes for sqlite.

I don’t think I've pushed any of that code to github yet, but i'll definitely do it soon (I'm currently on vacation). And I just remember that the namespace was already `mormot.ai.*`, and I think I had started working on a complete set of ai-related units, similar to what you mentioned in your first post. See:

sh17 · 2026-06-18 19:23:51

OK, then maybe we should try to merge that somehow after your vacation

sh17 · 2026-07-05 16:07:20

@flydev Have you been able to publish the code on GitHub yet? I’d share mine with you after that.

ab · 2026-07-05 20:47:45

Also please contact JP https://github.com/joaopauloschuler

He is proposing some great AI-generated samples to the trunk and has tremendous knowledge and ideas about AI.

flydev · 2026-07-08 07:31:24

Hello,

@sh17 I will push what I have. Also, @ab is right, JP's neural-api is really great!

flydev · 2026-07-28 10:59:29

Hi all,

@sh17 sorry for the delay

Following up on my last post - the code goes public today: https://github.com/flydev-fr/mormot.ai

Two independent pieces under the mormot.ai.* namespace, both mormot2 only, no external dependencies:

1. LiteLLM SDK port (mormot.ai.litellm.*): Its an unified LLM client - one call, any provider.
2. pgvector / RAG stack: Vector search and RAG over pascal source code: TPgVector binary wire-format codec, a pascal source chunker, an embedding-client interface, hybrid retrieval blending cosine similarity and PostgreSQL tsvector, a RAG orchestrator, and three CLI programs, and an attempt to ingest mormot2 code and build a chatbot using ollama/devstral-2, actually working.

There is a full docs site covering both parts: https://mormot-ai.fsb.dev
(serving the published "docs" folder).

Fair warning: the code works but is not finished, and the docs track it closely rather than exactly. Usable, not perfect.

Bonus: this documentation site is built and served with Mormotsaurus, a small CLI tool for authoring and publishing static documentation sites - itself written with mormot2 mvc layer. Windows and Linux binaries are in the repo for now; it will get its own repository soon.

flydev · 2026-07-28 14:31:40

The docs site is live, it should be easier to read than the md files on github.
And a small note on what is Nanobot in mormot-ai-sdk/ex/nanobot-sdk, it's an example of an autonomous personal AI assistant, like openclaw - it can interact with your computer files, using browser, etc and show how to use the current mormot.ai sdk.

edwinsn · 2026-07-29 03:19:15

A promising proposal, @sh17.

@flydev, well done! Would be interesting to see and check the upcoming Mormotsaurus repository.

sh17 · 2026-07-29 05:32:51

Thanks @flydev for posting this; I’ll consolidate everything, and then we’ll see what the next steps are.

sh17 · 2026-07-29 06:34:26

@flydev could you please add a license at https://github.com/flydev-fr/mormot.ai/ … mot-ai-sdk?

flydev · 2026-07-29 14:36:57

Done. PS: I have no idea what I’m doing when it comes to licensing

sh17 · 2026-07-29 18:34:18

The same license as mormot2 is the best solution.

flydev · 2026-07-29 18:37:38

there:

The files in this directory are released under a disjunctive MPL 1.1/GPL 2.0/LGPL 2.1 three-license, matching the licensing terms of the mORMot 2 framework

flydev · 2026-09-04 10:37:28

@sh17 did I miss something about the "FPC only" note in the README file?

delphi:

Software version tested: 1.0 (2026-09-04 11:53:55)
Generated with: Delphi 12.3 Athens 64 bit Win compiler

Time elapsed for all tests: 2.53s
Performed 4 Sep 2026, 11:54:04 by badb on DESKTOP-5501FNL

Total assertions failed for all test suits:  0 / 892
! All tests passed successfully.

fpc:

Software version tested: 1.0 (2026-09-04 11:53:52)
Generated with: Free Pascal 3.2.3 64 bit Win compiler

Time elapsed for all tests: 2.46s
Performed 4 Sep 2026, 11:54:01 by badb on DESKTOP-5501FNL

Total assertions failed for all test suits:  0 / 892
! All tests passed successfully.

I had to fix a small inconsistent check (ansi vs utf8).

sh17 · 2026-09-04 10:51:10

It seems to be working for the time being. However, I will no longer make allowances for Delphi or implement any special handling.

flydev · 2026-09-04 11:20:10

Understood, thanks.

sh17 · 2026-09-04 11:54:52

Do you need Delphi?

flydev · 2026-09-09 07:38:07

yes and no. on my side I use fpc only to compile headless and cross-platform. but delphi stays the everyday IDE here for debugging - for me lazarus is terrible.

edit: I must admit I always look at this through the mORMot 2 prism, as it supports both worlds

Last edited by flydev (2026-09-09 08:17:57)

sh17 · 2026-09-09 08:26:41

Okay, I’ll try to remain Delphi-compatible for as long as possible and remove any FPC-only references for now. The reason is that I’ll be using https://github.com/joaopauloschuler/neural-api in places, and that is FPC-only—but we’ll see.

sh17 · 2026-10-14 17:51:11

mormot2-ai - progress update (October 2026)

A short update on where the mormot.ai.* extension stands since the summer. Repository and README: github.com/landrix/mormot2-ai (same three-license as mORMot 2).

Where it lives
Arnaud, thanks for discussing a possible inclusion of mormot.ai.* in the framework. We decided to maintain it as a separate extension built on mORMot 2, with its own roadmap (own inference engine, later CUDA). mORMot 2 remains the foundation: HTTP server, JSON, RTTI, SQLite and crypto are all yours.

What it does today

MCP server, revision 2026-07-28 (the new stateless one). Tools are typed records with the input schema generated from RTTI. Transports: stdio, HTTP, Streamable HTTP, in-process. Optional OAuth 2.1 resource server. Our own application (Landrix) already uses it as its MCP endpoint.
LLM clients: the OpenAI wire (OpenAI, LiteLLM, Ollama-compatible) plus a native Anthropic driver. Streaming, tool calls, structured output, vision, and an agent/tool-calling loop that can use MCP tools.
Embeddings, vector search, RAG: one IEmbedder seam. A native vector store inside mORMot's SQLite (float32 rows, 1-bit RAM index with exact rescoring) replaced sqlite-vec; measured at 2 million x 1024 vectors.
Our own inference engine in plain Object Pascal, new since the last post. It runs BGE-M3 embeddings from the same GGUF Q8_0 file that sqlite-lembed (llama.cpp) uses, with the same quality: nDCG@10 0.9004 vs 0.8963, cosine >= 0.999 against lembed's vectors. It needs no Python, no llama.cpp and no foreign .so/.dll. The pieces: GGUF and safetensors loaders, a SentencePiece/Unigram tokenizer, the XLM-RoBERTa encoder, NEON and AVX2 kernels, and a dedicated spinning thread pool for the few hundred short parallel steps per text (TSynParallelProcess has a 1 ms sleep in its acquisition retry path, which adds up at that rate).

Speed, same corpus (320 texts), same model file, one text at a time:

Core i9-14900K, AVX2: native 1039 tokens/s vs lembed 524. Peak memory 377 vs 514 MB.
Snapdragon X, NEON: on par to 15% faster than lembed, with less memory.

Side finding on Windows 11: started over SSH, both engines ran at about half speed on that machine; switching off execution-speed power throttling for the process with SetProcessInformation(ProcessPowerThrottling) restored the numbers above.

Where it runs
Same compiler range as mORMot 2: FPC 3.2.2 / 3.2.4 and Delphi 7 to 13. I said in September that I would drop Delphi. I changed my mind, and it is now first class: the core is non-generic, generics are only an optional shell, and there is no exit(value) and no with. The test suites pass on:

Linux aarch64 (FPC): MCP, LLM and inference
Windows x64 (FPC and Delphi 13): MCP, LLM and inference
Windows x86 (Delphi 7): MCP, LLM and inference
macOS aarch64 (FPC): MCP and LLM
Windows ARM64 (FPC 3.3.1, with an external sqlite3-64.dll): MCP and LLM

Next: the inference suite on Linux x86_64 (our production target, Docker), macOS and Windows ARM64.

Credits

flydev: the MCP server units were adopted from your mormot2-extensions, original headers kept, and your commit is pinned for later diffs. Thank you.
JP, following Arnaud's pointer: we evaluated neural-api for the embeddings; the version we looked at already brought a lot of encoder and tokenizer support. Our specific needs - BGE-M3 from GGUF with its charsmap normalisation, NEON, concurrent callers, and every mORMot 2 compiler including Delphi 7 - led us to a dedicated inference engine instead.

What is planned

Native embeddings as our default (after the Linux x86_64 run and a long indexing run).
Decoder LLMs: Qwen3-0.6B first (BPE, KV cache, GQA) behind the same ILlmClient interface as the cloud providers, with streaming and tool calls.
Q4 / K-quants and sampling.
CUDA through the driver API with embedded PTX.

Feedback, issues and questions are welcome, in particular from anyone who wants to run it on a platform we have not listed.

edwinsn · 2026-10-15 05:33:52

@sh17, well done! Great achievement

ab · 2026-10-15 17:48:44

This is impressive work, and in very good shape.

I pushed today two small commits about SQLite3, which would remove some of your concern:
- https://github.com/synopse/mORMot2/commit/6302d231e
- https://github.com/synopse/mORMot2/commit/f08998d82

Feedback is welcome!
I guess there are some other thinks to port back to mORMot, perhaps some low level computation primitives (mormot.core.numbers.pas?) and e.g. https://github.com/synopse/mORMot2/issues/648 and some of your unicode stuff...

I am looking forward all this, especially the ILlmCLient native execution with good enough speed with a stand-alone executable.
That would be amazing. Why CUDA? It is NVidia only AFAIR - and I have a B580 board. Vulkan seems like a good candidate and we have good port thanks to PasVulkan via Bero.

sh17 · 2026-10-15 20:27:39

Thanks a lot, Arnaud!

SQLite3: both commits replace exactly the two workarounds we carried, and we already switched to them.

Our RollbackIfOpen existed because a failed Commit (e.g. SQLITE_BUSY) cleared the flag, so the RollBack afterwards did nothing. It is now a plain non-raising RollBack, and a new regression test checks it: a COMMIT fails against another connection's read lock, then RollBack closes the transaction.
Our own db_config/load_extension helpers are gone. Your LoadExtension is better, since it restores the previous state and never enables the SQL function.

One small observation: LoadExtension takes a TFileName. On non-Unicode compilers (Delphi 7, FPC on Windows), a path with characters outside the ANSI code page is lost before it reaches UTF-8, so a RawUtf8 overload would avoid that. It is not urgent for us, as those extensions will go once our native engine fully replaces them.

GPU: Vulkan first, CUDA second. Vulkan runs on NVIDIA, AMD, Intel (your B580), Adreno and, through MoltenVK, Apple.

On speed, llama.cpp's community benchmarks (indicative only) show token generation on NVIDIA about equal to CUDA, and prompt processing at about 70-95 % of CUDA. For example, an RTX 5090 does 11.8k vs 15.0k tokens/s pp512 and 274 vs 300 tokens/s tg128. So we keep CUDA as a second backend for maximum prompt throughput on NVIDIA.
For the binding we would use only PasVulkan's stand-alone Vulkan.pas header (MIT-style licence, Delphi 7 and FPC). It loads the system Vulkan loader dynamically, so we ship no DLL, and without a GPU the executable still runs on the CPU.
The kernels would be compute shaders compiled offline to SPIR-V and embedded in the executable; the MIT-licensed ggml-vulkan shaders are a good source.

flydev · 2026-10-18 14:37:25

I've done some testing, the goal was for a hobby project, to run a TTS model natively in delphi, no python at all. Model is Qwen3-TTS-12Hz-0.6B-CustomVoice. The setup:
delphi 13, mormot 2.4 trunk, Win64, win11 and an AMD RX 9060 XT gpu through Vulkan.

What I reused from mormot2-ai:
- safetensors loader, Qwen2 BPE tokenizer, CPU backend (AVX2 kernels + thread pool), sampler
- the Vulkan backend: Q8_0 linear shaders, plus rmsnorm, qkrope, attention, swiglu and addnorm, recorded the way TQwen3Vulkan.Chunk does it

What I wrote on top:
- the talker and the code predictor - Qwen3 stacks fed with embeddings instead of token ids
- the 12 Hz audio decoder (causal convolutions, SnakeBeta, sliding-window transformer)
- frame-by-frame streaming to waveOut

The 12 Hz decoder runs on the GPU as well, with four small compute shaders of my own (a tiled conv-as-GEMM with fused epilogues, SnakeBeta, depthwise conv, LayerNorm), built on the TInferVulkan batch API. It is x29 real time on its own, and its wav stays within 5e-6 of PyTorch.

Correctness, checked layer by layer against the official python package (qwen_tts, PyTorch, float32, greedy):
- the decoder wav matches within 2.5e-6
- in FP32 on CPU the generated codes are identical to PyTorch on 4 test sentences (french, english, accents, a 12s text)

Perf, same machine and same model:

 
                 python bridge (ROCm)   mormot2-ai, native
speed            x0.8-1.2 real time     x5.9 real time
first sound      ~2 s                   ~95 ms
model load       -                      ~1.4 s
VRAM             -                      ~0.9 GB (Q8_0 talker + F32 decoder)
on disk          ~9.5 GB of pytorch     nothing but the 2.4 GB model

Talker and code predictor run on Vulkan in Q8_0, about 12 ms per 80 ms audio frame. The CPU-only fallback (Q8_0) still runs the talker at x3.3 real time.

A few things I ran into, just in case it helps someone:
- InferRowToF32 doesn't read BF16, which is the format of the TTS weights - I widened the rows myself
- TBpeTokenizer only loads tokenizer.json. This model ships vocab.json + merges.txt, so I subclassed it and called the protected Build
- TQwen3Vulkan only accepts token ids. An entry point taking embeddings would make it reusable for TTS and multimodal models - for now I just copied the push-constant records to rebuild the same pass
- with the KV cache in F16 my checks drifted by ~2.3e-2 relative, against 2.6e-5 in F32. I keep F32, the context is short anyway
- a TInferThreadPool work item that allocates on the heap gets serialized by the memory manager - a 22-thread Q8 conversion took 3.5s instead of 0.6s, a stack buffer fixed it

great work @sh17

edit: a few precisions, the 12 Hz decoder is not entirely on the GPU: the quantizer, the pre-conv and the sliding-window transformer stay on the CPU, only the heavy part (upsampling, ConvNeXt and the convolution blocks) runs on the GPU, and
- decoder wav vs PyTorch: within 2.5e-6 on the CPU, 4.8e-6 on the GPU

Last edited by flydev (2026-10-18 14:48:27)

sh17 · 2026-10-18 15:31:13

@flydev - thanks a lot. A native TTS
pipeline at x5.9 real time with ~95 ms to first sound is impressive, and it is good to see the
loader/tokenizer/backends hold up outside the Qwen3 path they were written for.

I went through your points against the code; all of them are now in the backlog:

1. BF16 in InferRowToF32 - correct, it only knows F32/Q8_0/Q4_K/Q6_K today (the decoder widens
BF16 on its own path). BF16 and F16 will be added there.

2. TBpeTokenizer and vocab.json + merges.txt - correct, only tokenizer.json and GGUF are
supported. A third constructor for the GPT-2 style file pair will feed the same Build, so
no more subclassing; the pre-tokenizer then has to be chosen explicitly (GPT-2 by default),
since those two files do not carry it.

3. Embeddings as input instead of token ids - the most important one, and it will come first.
The plan is a forward that takes an embedding matrix (optionally mixed with ids inside one
chunk), with the same positions/KV handling, on CPU, Vulkan and CUDA alike. TTS, vision and
audio models all need it, and you should not have to copy the push-constant records.

4. F16 KV cache - yes, that is the expected trade-off (it is on by default for long contexts
and can be switched off). I will document the cost more clearly and consider defaulting to
F32 for short contexts, where autoregressive loops like your talker/predictor accumulate
the error.

5. Heap allocation in pool work items - the library's own jobs do not allocate inside the
items, but you are right that the API makes the safe pattern hard: a TOnInferItem does not
know its worker, so it cannot use a per-thread scratch buffer. I plan a variant that passes
the worker index (or a per-worker scratch area kept by the pool), plus a note in the unit
that items must not allocate.

I will post here when the first of these is pushed. If your TTS code ever becomes public,
I would be glad to add a pointer to it - and if you have a small reference (input embeddings
-> logits) from your model, it would make a nice regression test for point 3.

flydev · 2026-10-18 15:51:22

great. I will make a standalone sample and will publish it on github this evening or tomorrow on the morning

flydev · 2026-10-18 16:45:51

the code is public now: https://github.com/flydev-fr/mormot2-qwen-tts

the reference you asked for is in tests/data/infer-ref/qwen3-tts-lm, in your infer-ref format. It has embeddings in and logits out for both Qwen3 stacks of the model, computed by PyTorch in float32, so you can drop the folder into mormot2-ai as is.

one more finding: the Vulkan backend caches weights by host address, so weights freed and reallocated at the same address come back stale.

sh17 · 2026-10-19 06:08:26

@flydev - thank you, that was quick. The qwen3-tts-lm dataset is now in mormot2-ai
(tests/data/infer-ref/qwen3-tts-lm, unchanged from your commit 8f5a846, with your two
generator scripts under tools/infer-ref/qwen3-tts and the provenance in UPSTREAM_BASE and
NOTICE). It will be the regression test for the embeddings-input forward: both stacks
(talker and code predictor), layer states and logits, on CPU, Vulkan and CUDA.

You are right about the weight cache: both GPU backends look weights up by host address,
type and size, so a buffer freed and reallocated at the same address is served stale. The
unit header documents it as a contract, but it is a trap as soon as several stacks or
changing weights share one backend - exactly your case. It is now in the backlog next to
the embeddings input: a way to forget a weight or an address range when its owner frees
it, instead of ClearWeights for everything, on Vulkan and CUDA alike, with a test that
reuses the same address.

And thanks for publishing mormot2-qwen-tts. I will report back here when it is pushed.

flydev · 2026-10-19 13:18:51

Kokoro-82M now runs natively too, in the same repo: https://github.com/flydev-fr/mormot2-qwen-tts

it is much lighter than qwen3-tts and runs on the CPU only, about 30x real time, with the first sound after ~56 ms. The phonemes come from espeak-ng, loaded at runtime. I checked every stage against the PyTorch reference. The README has the details - feel free to check it.

sh17 · 2026-10-20 03:35:32

Impressive. If you bump the submodule and something is still missing for the TTS side, please tell me. Here is a summary of the changes since the last pin in your project.

- TDecoderModel.ForwardEmbeds - a forward pass with embedding rows instead of token ids, on the CPU, Vulkan and CUDA. Your talker and code predictor can feed their rows directly. This should make the copied pass in mormot.ai.tts.qwen.lm.vulkan.pas unnecessary. Your qwen3-tts-lm reference runs as a test (talker states within 1.8e-4 of PyTorch, the predictor codes are identical).
- TDecoderModel.CreateStack - loads one decoder stack out of a larger checkpoint by its prefix (talker and code predictor of Qwen3-TTS).
- ForgetWeights on the backend and on the model - this fixes the stale GPU weights you reported. New rule: free a model before its backend.
- TBpeTokenizer.CreateFromVocabMerges for vocab.json + merges.txt. It accepts only what transformers 5.19 loads with the same ids; for Qwen3-TTS it gives the reference ids.
- BF16 and F16 rows in InferRowToF32.
- TInferThreadPool.RunWorkers passes the worker index, so a job can keep its own scratch buffer instead of allocating. For you, allocating inside a job only cost time under Delphi (FPC's heap works per thread).
- The KvF16 docs now say when to turn it off. A talker loop that feeds floats back into the model is one such case: that is where your 2.3e-2 came from.

I also added gpt-oss on the CPU (attention sinks, YaRN, biased and clamped experts). MXFP4 weights and the GPU paths are next.

flydev · 2026-10-20 09:07:56

Thanks. I opened an issue on your repo, I think it will be better to discuss findings and issues there.

mORMot Open Source

#1 2026-06-17 21:03:11

Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#2 2026-06-18 10:08:53

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#3 2026-06-18 11:00:30

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#4 2026-06-18 11:51:46

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#5 2026-06-18 18:04:22

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#6 2026-06-18 18:42:52

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#7 2026-06-18 18:49:18

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#8 2026-06-18 19:15:27

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#9 2026-06-18 19:23:51

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#10 2026-07-05 16:07:20

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#11 2026-07-05 20:47:45

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#12 2026-07-08 07:31:24

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#13 2026-07-28 10:59:29

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#14 2026-07-28 14:31:40

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#15 2026-07-29 03:19:15

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#16 2026-07-29 05:32:51

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#17 2026-07-29 06:34:26

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#18 2026-07-29 14:36:57

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#19 2026-07-29 18:34:18

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#20 2026-07-29 18:37:38

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#21 2026-09-04 10:37:28

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#22 2026-09-04 10:51:10

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#23 2026-09-04 11:20:10

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#24 2026-09-04 11:54:52

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#25 2026-09-09 07:38:07

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#26 2026-09-09 08:26:41

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#27 2026-10-14 17:51:11

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#28 2026-10-15 05:33:52

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#29 2026-10-15 17:48:44

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#30 2026-10-15 20:27:39

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#31 2026-10-18 14:37:25

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#32 2026-10-18 15:31:13

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#33 2026-10-18 15:51:22

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#34 2026-10-18 16:45:51

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#35 2026-10-19 06:08:26

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#36 2026-10-19 13:18:51

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#37 2026-10-20 03:35:32

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

#38 2026-10-20 09:07:56

Re: Proposing a `mormot.ai.*` extension — Model Context Protocol (MCP)

Board footer