sūveryn › Benchmark
Your data. Your premises. Your AI.
In November 2026 we ran the whole product on real contracts and deeds: reading them, answering 106 questions, and checking every answer against the passages it cites. 93 of 106 answers passed every check, in seconds each, on hardware an office can own. The reports and the code are public, so you can check it yourself.
Measured 11 to 13 November 2026 · published 15 November 2026
The documents are real, used with their owner’s permission on a rented GPU machine and deleted afterwards. The report holds counts only: no document, party or sentence is named.
| Documents | Eleven: ten PDFs of 3 to 98 pages (299 pages in all, 145 of them scans) and one Word file |
|---|---|
| Kind | Mostly company contracts, with a few notarial deeds, in Dutch, French and English, from one owner |
| Questions | 106, in 23 sets, mostly in Dutch |
| Models | Qwen3.8-27B (4-bit, llama.cpp); Mistral Small 3.2 24B for comparison. Both are open-weight and run on the server itself |
| Graphics card | An RTX 4090 with 24 GB, the minimum sūveryn asks for; two rounds on cards with 32 GB |
| Code | suveryn-core as it stood each day, with its own answers test |
The same documents and questions on three days, with Qwen3.8-27B. What the first round showed went into the code before the second. Between the rounds both the code and the card changed, so the rise cannot be put down to one of them.
| 11 Nov, 32 GB card | 12 Nov, RTX 4090 (24 GB) | 13 Nov, RTX 5090 (32 GB) | |
|---|---|---|---|
| Answers that pass | 83 of 106 | 88 of 106 | 93 of 106 |
| Sums made by the server | 5 of 8 | 7 of 8 | 7 of 8 |
| Seconds per answer | 13.6 | 11.6 | 7.4 |
An answer counts as passed only when all five hold:
These are fixed rules that give the same result every time, not a second opinion from a model. Where a rule finds something, sūveryn flags it for the reader, and a person decides.
A page of digital text takes 0.1 to 0.4 seconds, a scanned page 0.4 to 1.6. The longest document, 98 pages of which 66 are scans, was read in 34 seconds on the 32 GB card and in 56 seconds on the RTX 4090. Every document was read without an error.
For 44 of the 106 questions the documents were longer than the model reads at once, and the answer was made from the passages that match the question best.
12 November, on an RTX 4090 with 24 GB.
| Mistral Small 3.2 24B | Qwen3.8-27B | |
|---|---|---|
| Answers that pass | 85 of 106 | 88 of 106 |
| Items the checks report | 108 | 63 |
| Sums made by the server | 6 of 8 | 7 of 8 |
| Seconds per answer | 8.8 | 11.6 |
| Room for the document reader on a 24 GB card | Needs a larger card | Yes |
Qwen is the default: more of its answers pass, and it leaves room for the document reader on the minimum card. Mistral Small, from the European lab Mistral AI, answers faster. With the settings tried so far it needs a larger card than 24 GB, so it stays a second model for bigger machines.
In October we measured the whole pipeline on one RTX 4090: reading scans, the two models, several people at once and documents of 60 pages. The test documents were synthetic deeds and three real documents, two of them Belgian notarial deeds. Qwen made fewer legal errors on the real deeds; Mistral was 20 to 40% faster. The report gives the hardware we recommend for an office.
Everything here comes from three public reports, and the answers test is part of the open-source code. On a GPU machine, with your own documents and a file of questions beside each:
OWN_QUESTIONS=/path/to/your/documents bash scripts/gpu-round.sh
In this test, for most of them: 93 of 106 answers passed every check in the latest round, at 7 to 12 seconds per answer depending on the card. Every answer cites its passages, fixed rules check them, and where a rule finds something sūveryn flags it, so a person can look at exactly that passage.
Qwen3.8-27B. On a 24 GB card 88 of its 106 answers passed, against 85 for Mistral Small 3.2 24B, with fewer items reported by the checks, and it leaves room for the document reader. Mistral answers faster and stays available as a second model for a larger card.
A server on your own premises with at least an NVIDIA GeForce RTX 4090 (24 GB): the whole product ran on that card with no memory error. For an office machine the hardware benchmark specifies 64 GB of system memory.
The documents were real contracts and deeds, used with their owner’s permission on a rented GPU machine and deleted afterwards. The reports hold counts only: no document, party or sentence is named.
A pilot runs on closed files, on a server in your own office. You check the answers against documents you know.
We reply within two working days.
Sūveryn for other practices