sūveryn › Benchmark

Your data. Your premises. Your AI.

Benchmark: local AI on real contracts and deeds, measured on one graphics card

In November 2026 we ran the whole product on real contracts and deeds: reading them, answering 106 questions, and checking every answer against the passages it cites. 93 of 106 answers passed every check, in seconds each, on hardware an office can own. The reports and the code are public, so you can check it yourself.

Measured 11 to 13 November 2026 · published 15 November 2026

  • 93 of 106answers pass every check in the latest round; 83 and 88 in the two rounds before
  • 299 pagesin ten PDFs, 145 of them scans, all read without an error
  • 24 GBone RTX 4090 ran the whole product, with no memory error
  • 11.6 sper answer on an RTX 4090; 7.4 s on an RTX 5090

What was tested

The documents are real, used with their owner’s permission on a rented GPU machine and deleted afterwards. The report holds counts only: no document, party or sentence is named.

DocumentsEleven: ten PDFs of 3 to 98 pages (299 pages in all, 145 of them scans) and one Word file
KindMostly company contracts, with a few notarial deeds, in Dutch, French and English, from one owner
Questions106, in 23 sets, mostly in Dutch
ModelsQwen3.8-27B (4-bit, llama.cpp); Mistral Small 3.2 24B for comparison. Both are open-weight and run on the server itself
Graphics cardAn RTX 4090 with 24 GB, the minimum sūveryn asks for; two rounds on cards with 32 GB
Codesuveryn-core as it stood each day, with its own answers test

The answers, in three rounds

The same documents and questions on three days, with Qwen3.8-27B. What the first round showed went into the code before the second. Between the rounds both the code and the card changed, so the rise cannot be put down to one of them.

11 Nov, 32 GB card12 Nov, RTX 4090 (24 GB)13 Nov, RTX 5090 (32 GB)
Answers that pass83 of 10688 of 10693 of 106
Sums made by the server5 of 87 of 87 of 8
Seconds per answer13.611.67.4

Every answer is checked by five rules

An answer counts as passed only when all five hold:

These are fixed rules that give the same result every time, not a second opinion from a model. Where a rule finds something, sūveryn flags it for the reader, and a person decides.

Reading the documents

A page of digital text takes 0.1 to 0.4 seconds, a scanned page 0.4 to 1.6. The longest document, 98 pages of which 66 are scans, was read in 34 seconds on the 32 GB card and in 56 seconds on the RTX 4090. Every document was read without an error.

For 44 of the 106 questions the documents were longer than the model reads at once, and the answer was made from the passages that match the question best.

Two open models, both on your own server

12 November, on an RTX 4090 with 24 GB.

Mistral Small 3.2 24BQwen3.8-27B
Answers that pass85 of 10688 of 106
Items the checks report10863
Sums made by the server6 of 87 of 8
Seconds per answer8.811.6
Room for the document reader on a 24 GB cardNeeds a larger cardYes

Qwen is the default: more of its answers pass, and it leaves room for the document reader on the minimum card. Mistral Small, from the European lab Mistral AI, answers faster. With the settings tried so far it needs a larger card than 24 GB, so it stays a second model for bigger machines.

What the test changed in the product

What this round does not cover yet

The earlier round: hardware, October 2026

In October we measured the whole pipeline on one RTX 4090: reading scans, the two models, several people at once and documents of 60 pages. The test documents were synthetic deeds and three real documents, two of them Belgian notarial deeds. Qwen made fewer legal errors on the real deeds; Mistral was 20 to 40% faster. The report gives the hardware we recommend for an office.

Hardware benchmark, October 2026

Read it, run it

Everything here comes from three public reports, and the answers test is part of the open-source code. On a GPU machine, with your own documents and a file of questions beside each:

OWN_QUESTIONS=/path/to/your/documents bash scripts/gpu-round.sh

Frequently asked questions

Can an open-weight model on one graphics card answer questions about real contracts?

In this test, for most of them: 93 of 106 answers passed every check in the latest round, at 7 to 12 seconds per answer depending on the card. Every answer cites its passages, fixed rules check them, and where a rule finds something sūveryn flags it, so a person can look at exactly that passage.

Which model does sūveryn use by default, and why?

Qwen3.8-27B. On a 24 GB card 88 of its 106 answers passed, against 85 for Mistral Small 3.2 24B, with fewer items reported by the checks, and it leaves room for the document reader. Mistral answers faster and stays available as a second model for a larger card.

What hardware does this need?

A server on your own premises with at least an NVIDIA GeForce RTX 4090 (24 GB): the whole product ran on that card with no memory error. For an office machine the hardware benchmark specifies 64 GB of system memory.

Were client documents used for the test?

The documents were real contracts and deeds, used with their owner’s permission on a rented GPU machine and deleted afterwards. The reports hold counts only: no document, party or sentence is named.

Judge it on a file you know by heart

A pilot runs on closed files, on a server in your own office. You check the answers against documents you know.

We reply within two working days.

Sūveryn for other practices