Product study · October 2026
Which model for which job? Eighteen models, tested inside a governed product.
In October 2026 we put eighteen large language models from five providers through three jobs our product does every day: writing the descriptions in a Business Overlay’s catalogue, producing a CFO agent’s nightly briefing, and answering a finance question in chat. Every run used the product’s real code paths, the same synthetic revenue model and the same language, Turkish. Two judges from different vendors, neither of them a contestant, scored the outputs without seeing which model wrote them. No model won all three jobs. Models served from inside Türkiye came second in two of them. And in chat, the engine that checks every number removed 107 figures the models had written but could not trace to a query result.
Ali Yıldırım, Founder and CEO, D-CAT Group · 10 October 2026
At a glance
No model won all three jobs
Scores run from 1 to 5 and average both judges. With two runs per model, differences smaller than about 0.3 points cannot be told apart, so read each list as a leading group, not a strict order. Costs are real token counts at each provider’s list price on the test date.
Job 1 · 745 catalogue objects
Catalogue descriptions
GPT-5.6 SolOpenAI · 4.18
GPT-5.5OpenAI · 4.12
GLM-5.3EVREN · 3.82
First and second are a tie. Third is shared, within 0.1, with GLM-5.3 at Zhipu, GPT-5.6 Terra and GPT-5.6 Luna.
Best paid valueGPT-5.6 Luna: 3.73, the whole catalogue for $0.08
Best operated in TürkiyeGLM-5.3 at EVREN: 3.82
Job 2 · 2 runs per model
Nightly CFO briefing
Claude Sonnet 4.6Anthropic · 4.67
GLM-5.3EVREN · 4.50
GLM-5.3Zhipu · 4.42
The first three sit within 0.3 of each other. Claude Opus 5.5 and DeepSeek V4.1 Flash at EVREN follow just behind, at 4.33.
Best paid valueGLM-5.3 at Zhipu: 4.42 for about $0.29 a briefing, against $1.82 for the leader
Best operated in TürkiyeGLM-5.3 at EVREN: 4.50
Job 3 · 2 conversations per model
Chat answer
Claude Opus 5.5Anthropic · 4.54
DeepSeek V4.1 FlashEVREN · 4.21
Claude Sonnet 4.6Anthropic · 3.83
Opus 5.5 leads by 0.33, at the edge of what two conversations can separate. The gap from second to third is clearer.
Best paid valueGLM-5.3 at Zhipu: 3.71 for about $0.09 a conversation, against $0.50 for the leader
Best operated in TürkiyeDeepSeek V4.1 Flash at EVREN: 4.21
EVREN’s models were free during the test (until 1 November 2026), so they are left out of the value line: a price of zero says nothing about what they will cost. LLMaaService’s best model, Qwen 3.8 27B, did not produce a briefing in either run.
Method
How we tested
Each model did the product’s own work through the product’s own code, with the same inputs. Nothing was hand-tuned for one model.
- The data
- A synthetic revenue model built for testing: an organisation’s revenue model on SQL Server Analysis Services, with about 80 tables and 745 catalogue objects (tables, columns, measures and metrics). It contains no personal data, and no figure from it is reported on this page.
- The models
- Twelve first: Claude Opus 5.5 and Claude Sonnet 4.6 (Anthropic); GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna and GPT-5.5 (OpenAI); GLM-5.3 (Zhipu); and five open-weight models from LLMaaService, an LLM service operated in Türkiye: Qwen 3.8 27B, gpt-oss-120b, Gemma 4 26B, Qwen 3.6 35B-A3B and Qwen3-Next 80B. Then six from EVREN (Savunma Sanayi Yapay Zeka EVREN), a Turkish defence-industry AI service, independent of LLMaaService: GLM-5.3, DeepSeek V4.1 Flash, DeepSeek V4 Flash, MiMo V2.6 Pro, Qwen 3.8 Flash Next and Gemma 4 31B.
- Job 1: descriptions
- Each model wrote a Turkish description for every one of the 745 objects, and for every measure whether a higher value is better. Every model received the same 21 requests; the EVREN models received the byte-identical saved requests. Only catalogue metadata was sent: names, types, row counts, relationships and measure formulas. No row values were sent. The product’s validator checked every answer. Quality was rated on a sample of 60 objects (20 tables, 20 columns, 20 measures).
- Job 2: CFO briefing
- The product’s CFO agent on this model, started through the real agent path with its own budgets and a limit of 20 steps a run, as an experiment run: nothing was sent to anyone and the results stayed hidden from users. Two runs per model. Before a finding or recommendation could publish, the engine checked every number in it against the query results it came from.
- Job 3: chat
- A real question asked earlier in the product, “How did VAT-inclusive revenue trend monthly over the last 6 months?”, then a follow-up, “Interpret these results, and run new queries if needed.” Two separate conversations per model, through the product’s real chat path. As in normal operation, the model saw the aggregated results of its own queries. It never held a credential or a path to the data.
- Reasoning setting
- Every model ran at the product’s default setting, Standard, which switches reasoning off wherever a model allows it. GLM-5.3 and gpt-oss-120b cannot switch it off and ran at their lowest level, “low”. Configurations with reasoning turned on were not tested, and they could change the picture, particularly for the OpenAI and Anthropic models.
- Judging
- Two judges that were not contestants: Claude Sonnet 5.5 (Anthropic) and gpt-5.4 (OpenAI). Model names were hidden and the outputs shuffled. The judges ranked the models almost identically: rank correlation 0.95 on descriptions, 0.96 on the briefing and 0.97 on chat. Neither visibly favoured its own vendor: on descriptions the Anthropic judge put two OpenAI models first and second, and in chat the OpenAI judge put Claude Opus 5.5 first.
- Scoring
- Descriptions: the mean of accuracy and specificity across both judges; fabrication is reported separately. Briefing and chat: the mean of insight, clarity and actionability across both judges and every run. A run that published nothing scored 1, the lowest possible, on every dimension.
- Cost
- Real token counts multiplied by each provider’s list price on the test date. EVREN was free during the test. Judging is not included. The scored outputs cost about $24 at list prices across Anthropic, OpenAI and Zhipu combined.
- Left out on purpose
- Claude Fable 5.1 will be tested separately; we will share the results in the coming days.
The eighteen models, as tested
| Model | Provider | List price per 1M tokens, input / output | Reasoning at the Standard setting |
|---|---|---|---|
| Claude Opus 5.5 | Anthropic | $4 / $20 | Off |
| Claude Sonnet 4.6 | Anthropic | $3 / $15 | Off |
| GPT-5.6 Sol | OpenAI | $4 / $20 | Off (stated as “none”) |
| GPT-5.6 Terra | OpenAI | $2 / $12 | Off (stated as “none”) |
| GPT-5.6 Luna | OpenAI | $0.2 / $1.2 | Off (stated as “none”) |
| GPT-5.5 | OpenAI | $5 / $30 | Off (stated as “none”) |
| GLM-5.3 | Zhipu | $1.4 / $4.4 | Low (cannot be switched off) |
| Qwen 3.8 27B | LLMaaService | $0.61 / $3.2 | Off |
| gpt-oss-120b | LLMaaService | $0.61 / $3.2 | Low (cannot be switched off) |
| Gemma 4 26B | LLMaaService | $0.4 / $2.2 | Off (model default) |
| Qwen 3.6 35B-A3B | LLMaaService | $0.55 / $2.4 | Off |
| Qwen3-Next 80B | LLMaaService | $0.61 / $3.2 | None (instruct model) |
| GLM-5.3 | EVREN | free until 1 Nov 2026 ¹ | Low (off for descriptions) |
| DeepSeek V4.1 Flash | EVREN | free until 1 Nov 2026 ¹ | Off |
| DeepSeek V4 Flash | EVREN | free until 1 Nov 2026 ¹ | Off |
| MiMo V2.6 Pro | EVREN | free until 1 Nov 2026 ¹ | Off |
| Qwen 3.8 Flash Next | EVREN | free until 1 Nov 2026 ¹ | Off |
| Gemma 4 31B | EVREN | free until 1 Nov 2026 ¹ | Off |
¹ EVREN states its prices in its own credits and showed every model as free until 1 November 2026.
Job 1 · Catalogue descriptions
OpenAI models wrote the most accurate descriptions and invented the least
GPT-5.6 Sol and GPT-5.5 tie at the top. Four models follow within 0.1 of each other: GLM-5.3 at EVREN and at Zhipu, GPT-5.6 Terra and GPT-5.6 Luna. Luna described the whole catalogue for eight cents.
| Model | Provider | Score | Accuracy | Specificity | Fabrication flagged, judge A / B | Accepted first time | Median length, characters | Cost, 745 objects | Median seconds per request |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | OpenAI | 4.18 | 4.65 | 3.71 | 10% / 12% | 745 / 745 | 128 | $1.82 | 30 |
| GPT-5.5 | OpenAI | 4.12 | 4.63 | 3.62 | 8% / 17% | 745 / 745 | 130 | $2.57 | 28 |
| GLM-5.3 | EVREN | 3.82 | 4.21 | 3.42 | 25% / 35% | 740 / 745 | 98 | free during test | 78 |
| GLM-5.3 | Zhipu | 3.81 | 4.19 | 3.42 | 28% / 38% | 705 / 745 | 92 | $0.41 | 36 |
| GPT-5.6 Terra | OpenAI | 3.79 | 4.55 | 3.04 | 2% / 8% | 745 / 745 | 93 | $0.86 | 18 |
| GPT-5.6 Luna | OpenAI | 3.73 | 4.50 | 2.96 | 3% / 17% | 745 / 745 | 92 | $0.08 | 17 |
| Claude Opus 5.5 | Anthropic | 3.67 | 4.22 | 3.12 | 27% / 28% | 745 / 745 | 57 | $2.75 | 36 |
| DeepSeek V4.1 Flash | EVREN | 3.64 | 4.05 | 3.22 | 33% / 42% | 745 / 745 | 100 | free during test | 74 |
| Claude Sonnet 4.6 | Anthropic | 3.51 | 3.67 | 3.35 | 55% / 63% | 745 / 745 | 125 | $2.17 | 96 |
| Qwen 3.8 27B | LLMaaService | 3.32 | 3.69 | 2.96 | 48% / 52% | 608 / 745 | 82 | $0.30 | 145 |
| gpt-oss-120b | LLMaaService | 3.31 | 4.11 | 2.50 | 12% / 20% | 700 / 745 | 69 | $0.24 | 25 |
| DeepSeek V4 Flash | EVREN | 3.30 | 4.02 | 2.58 | 20% / 25% | 745 / 745 | 64 | free during test | 21 |
| MiMo V2.6 Pro | EVREN | 3.25 | 3.46 | 3.04 | 35% / 42% | 647 / 745 | 107 | free during test | 87 |
| Qwen 3.8 Flash Next | EVREN | 3.24 | 3.86 | 2.61 | 32% / 43% | 741 / 745 | 62 | free during test | 34 |
| Qwen 3.6 35B-A3B | LLMaaService | 3.01 | 3.56 | 2.45 | 45% / 58% | 744 / 745 | 73 | $0.26 | 35 |
| Qwen3-Next 80B | LLMaaService | 2.96 | 3.41 | 2.52 | 50% / 57% | 745 / 745 | 88 | $0.37 | 40 |
| Gemma 4 26B | LLMaaService | 2.93 | 3.99 | 1.88 | 12% / 18% | 720 / 745 | 51 | $0.15 | 22 |
| Gemma 4 31B | EVREN | 2.89 | 3.70 | 2.08 | 23% / 42% | 745 / 745 | 54 | free during test | 55 |
Judge A is Claude Sonnet 5.5, judge B is gpt-5.4. “Fabrication flagged” is the share of the 60 sampled descriptions in which that judge found a claim the metadata did not support. “Accepted first time” counts objects whose description the product’s validator accepted from the first round of requests; the rest are asked again.
- Fabrication separated the field more than fluency did. GPT-5.6 Terra was flagged in 2% and 8% of the sampled descriptions, GPT-5.6 Luna in 3% and 17%. Claude Sonnet 4.6 was flagged in 55% and 63%, the most of all eighteen. A typical case: a description of a category table that adds example category names appearing nowhere in the metadata.
- Saying little is not the same as being right. Gemma 4 26B and gpt-oss-120b invented little, but they also said little, and their specificity scores were among the lowest. Claude Opus 5.5 wrote the shortest descriptions of the top half.
- The same model at two gateways wrote the same quality. GLM-5.3 scored 3.82 at EVREN and 3.81 at Zhipu. EVREN’s requests took about twice as long.
Job 2 · Nightly CFO briefing
The briefing rewarded depth, and five models reached it
Claude Sonnet 4.6 leads, with GLM-5.3 at EVREN and at Zhipu within 0.3 of it. Claude Opus 5.5 and DeepSeek V4.1 Flash at EVREN follow just behind, at 4.33. Two of these five ran on a service operated in Türkiye. Three models produced no briefing at all.
| Model | Provider | Score | Runs that published findings | Findings and recommendations published, run 1 / run 2 | Cost per run | Minutes per run | Note |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | Anthropic | 4.67 | 2 of 2 | 10 / 10 | $1.82 | 5.6 | |
| GLM-5.3 | EVREN | 4.50 | 2 of 2 | 7 / 7 | free during test | 6.0 | |
| GLM-5.3 | Zhipu | 4.42 | 2 of 2 | 8 / 4 | $0.29 | 7.0 | |
| Claude Opus 5.5 | Anthropic | 4.33 | 2 of 2 | 8 / 9 | $2.16 | 5.3 | |
| DeepSeek V4.1 Flash | EVREN | 4.33 | 2 of 2 | 8 / 10 | free during test | 4.4 | |
| GPT-5.6 Sol | OpenAI | 3.92 | 2 of 2 | 10 / 10 | $0.41 | 1.6 | |
| GPT-5.6 Terra | OpenAI | 3.83 | 2 of 2 | 8 / 8 | $0.18 | 1.1 | |
| DeepSeek V4 Flash | EVREN | 3.75 | 2 of 2 | 5 / 7 | free during test | 2.8 | |
| GPT-5.6 Luna | OpenAI | 3.50 | 2 of 2 | 5 / 5 | $0.02 | 1.2 | |
| Gemma 4 26B | LLMaaService | 3.08 | 2 of 2 | 3 / 3 | $0.06 | 0.8 | |
| MiMo V2.6 Pro | EVREN | 2.83 | 2 of 2 | 6 / 3 | free during test | 9.0 | |
| Gemma 4 31B | EVREN | 2.75 | 2 of 2 | 6 / 4 | free during test | 1.6 | |
| GPT-5.5 | OpenAI | 2.67 | 1 of 2 | 0 / 9 | $0.41 | 1.3 | one of two runs published nothing |
| Qwen3-Next 80B | LLMaaService | 2.00 | 2 of 2 | 8 / 8 | $0.18 | 1.9 | |
| gpt-oss-120b | LLMaaService | 1.33 | 1 of 2 | 5 / 0 | $0.08 | 0.5 | one of two runs published nothing |
| Qwen 3.8 27B | LLMaaService | 1.00 | 0 of 2 | 0 / 0 | $1.23 | 1.4 | no briefing in either run |
| Qwen 3.6 35B-A3B | LLMaaService | 1.00 | 0 of 2 | 0 / 0 | $0.03 | 0.3 | no briefing in either run |
| Qwen 3.8 Flash Next | EVREN | 1.00 | 0 of 2 | 0 / 0 | free during test | 1.2 | no briefing in either run |
A run that published no finding or recommendation scored 1. Costs and times are the mean of the two runs.
- What the judges rewarded. In the top five they credited scepticism about the data itself, findings that connected several signals, and recommendations with a named owner and a threshold. Their most common complaint was density: long findings, and briefs that repeated them.
- Fast and clear, but less specific. The OpenAI models finished in one to two minutes, against four to seven for the top five, and wrote clearly. With reasoning off, their recommendations were less specific about owners and thresholds. GPT-5.5’s first run published nothing; its second scored 4.33.
- Where models failed. Qwen 3.8 27B at LLMaaService and Qwen 3.8 Flash Next at EVREN kept running queries until the 20-step limit and never wrote the briefing. Qwen 3.6 35B-A3B tried to answer without querying any data, which the engine does not accept, and the run ended. gpt-oss-120b’s second run published nothing. MiMo V2.6 Pro’s second run described an older month instead of the current one.
Job 3 · Chat answer
In chat, Claude Opus 5.5 led and a model served in Türkiye came second
Claude Opus 5.5 scored highest on both the first question and the follow-up. DeepSeek V4.1 Flash at EVREN came second, and did better on the follow-up, where the model has to decide which new queries to run, than on the first question.
| Model | Provider | Score | First question | Follow-up | Figures removed by the engine, 4 answers | Cost per conversation | Mean seconds per answer | Note |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Anthropic | 4.54 | 4.50 | 4.58 | 0 | $0.50 | 46 | |
| DeepSeek V4.1 Flash | EVREN | 4.21 | 4.08 | 4.33 | 0 | free during test | 48 | |
| Claude Sonnet 4.6 | Anthropic | 3.83 | 4.00 | 3.67 | 5 | $0.34 | 60 | |
| GLM-5.3 | Zhipu | 3.71 | 3.33 | 4.08 | 0 | $0.09 | 38 | |
| GLM-5.3 | EVREN | 3.71 | 3.67 | 3.75 | 0 | free during test | 63 | |
| Qwen 3.8 Flash Next | EVREN | 3.58 | 4.08 | 3.08 | 0 | free during test | 32 | 1 of 4 answers unfinished |
| Qwen 3.8 27B | LLMaaService | 3.42 | 2.83 | 4.00 | 5 | $0.22 | 40 | |
| GPT-5.6 Sol | OpenAI | 3.17 | 3.08 | 3.25 | 0 | $0.18 | 18 | |
| GPT-5.6 Luna | OpenAI | 3.17 | 2.67 | 3.67 | 0 | $0.01 | 16 | |
| DeepSeek V4 Flash | EVREN | 3.12 | 3.50 | 2.75 | 0 | free during test | 62 | 1 of 4 answers unfinished |
| GPT-5.5 | OpenAI | 2.96 | 2.83 | 3.08 | 0 | $0.16 | 12 | |
| GPT-5.6 Terra | OpenAI | 2.92 | 2.92 | 2.92 | 0 | $0.09 | 16 | |
| Gemma 4 26B | LLMaaService | 2.25 | 2.25 | 2.25 | 1 | $0.06 | 15 | |
| Qwen3-Next 80B | LLMaaService | 2.08 | 1.75 | 2.42 | 26 | $0.08 | 12 | |
| MiMo V2.6 Pro | EVREN | 2.08 | 3.17 | 1.00 | 0 | free during test | 69 | 2 of 4 answers unfinished |
| Gemma 4 31B | EVREN | 2.08 | 1.75 | 2.42 | 0 | free during test | 47 | |
| gpt-oss-120b | LLMaaService | 2.04 | 2.00 | 2.08 | 19 | $0.08 | 10 | |
| Qwen 3.6 35B-A3B | LLMaaService | 1.96 | 2.25 | 1.67 | 51 | $0.11 | 20 |
A conversation is the question and its follow-up. “Figures removed” counts numbers the model wrote that the engine could not trace to a query result and took out of the answer before it reached the reader.
- Clarity was not what separated the models. Most models scored between 3.6 and 4.4 on clarity. The differences came from insight and actionability.
- The quickest answers rarely said what to do next. The OpenAI models answered in 12 to 18 seconds on average, cleanly, but seldom proposed a next step: their actionability averaged 1.6 to 2.1, against 4.4 for Claude Opus 5.5. All of them ran with reasoning off.
- A few answers did not finish within the agent’s limits. Four of the 72, all on EVREN: two from MiMo V2.6 Pro and one each from DeepSeek V4 Flash and Qwen 3.8 Flash Next.
What the engine did meanwhile
The model writes the sentence. The engine checks every number in it.
Scores measure how good an answer reads. They do not show what a reader was protected from. The same checks ran for every model, and none of them depends on which model is plugged in.
Chat
107 figures removed
Numbers that models wrote but no query result carried, taken out of 11 of the 72 answers before anyone read them. 96 of them came from three open-weight models: Qwen 3.6 35B-A3B, Qwen3-Next 80B and gpt-oss-120b. Twelve models had none removed.
Briefing
31 drafts sent back
Across the 36 briefing runs, the engine returned a draft to the agent 31 times with a correction to make before it could go further. The rule is the same for every model.
Briefing
25 candidates held
219 findings, recommendations and briefs published. 25 candidate findings were held at the structural check on their claims, each with its reason on the record.
Engineering notes
Five things the test taught us about running many models
None of these is specific to our product. Anyone putting several models behind one application will meet most of them.
“Reasoning off” can move the reasoning into the answer. Asked to switch reasoning off, GLM-5.3 at EVREN kept reasoning and wrote it into the answer instead (“Wait… Hmm… Let me…”) until it hit the output limit. A capability check that reads only the separate reasoning field reports that switching off works. Left on, the model spent its whole output allowance on reasoning. The setting that worked was a reasoning level of “low” in the request body: short, clean answers in about 30 seconds. With that setting it completed every briefing and chat run in this test. Measure the answer, not the field.
The same model name can sit behind two gateways. GLM-5.3 is offered by Zhipu and by EVREN under the same identifier. A system that finds a model by its name alone will send traffic to the wrong provider and bill it at the wrong price. Key every model by provider and model together.
Reasoning switches differ by model, not by provider. One provider’s models used two different switches; some models cannot turn reasoning off at all (GLM-5.3, gpt-oss-120b); and the GPT-5.6 models accept tool calls only when a reasoning level is stated explicitly, even when that level is “none”. The only reliable source is a live measurement of each model.
Some models loop instead of concluding. Two Qwen-family models ran query after query until the 20-step limit and never wrote the briefing. Because the context grew with every step, one of the cheapest models in the test became the third most expensive in this job: $2.45 for two runs that published nothing. The step limit ended both runs.
Fluent is not the same as faithful. The description test produced readable text from every model. The difference was in what they added: the share of descriptions with an unsupported claim ranged from a few per cent to more than half. If a model writes documentation people will trust, measure fabrication on its own.
Stated before you ask
What this study does not show
- It is small. Two runs per model for the briefing and two conversations for chat. Differences under about 0.3 points are not significant, and a single bad run moves a model a long way.
- It is one model, one domain, one language. One synthetic revenue model, in Turkish. Results on another data model, industry or language may differ.
- The judges are models. They were blind, from two different vendors, and agreed closely, but they are not expert human reviewers, and both come from vendors whose other models were tested.
- It tested one setting. Reasoning was off wherever a model allowed it. With reasoning on, several models would cost more and could score differently.
- Prices and speeds are snapshots. EVREN was free during the test and we had no price for afterwards. On LLMaaService, the same model and the same requests took about 25 seconds a call in one session and about 145 seconds a few hours later. Providers also update their models.
- The data is synthetic. A revenue model built for testing, with no personal data. It is not a customer result.
Choose the model per job. Keep the checks that do not depend on the model.
For the buying team
What this means if you are choosing a model
Choose per job, not per vendor
The best description writer was mid-table in chat. Two models that answered chat reasonably well never produced a briefing. In AgenticObjects an administrator sets the approved model per agent, a separate model for chat, and the model for catalogue work, so each job can use what suits it.
An in-country option is now competitive
For organisations whose data must stay in Türkiye, GLM-5.3 and DeepSeek V4.1 Flash at EVREN came second in the briefing and in chat respectively. Weigh that against what is still open: a price after the free period, and answers that took longer than the fastest international models.
Keep the checks outside the model
Every model in this test had at least one draft sent back by the engine. A check that lives outside the model keeps working when you change the model. One model per run, chosen before it starts and written onto the record, so a change is a decision someone made, not something that happened.