A procurement-grade comparison of 10 large language models on real customer-support workflows. Same task, same retrieval, same prompt, same scoring rubric across every model. The default view is the workload-shaped operating recommendation; analyst-depth diagnostics sit one layer deeper.
Use a workload split, not a single model standard. Keep the fast production model for live chat. Use the strongest gate-passing model for asynchronous, high-risk, or quality-critical flows. Treat models with safety or completeness failures as not shortlisted unless a specific commercial constraint justifies the risk.
Best value (high composite, low spend per 560 queries) lives in the top-left quadrant. Bottom-right models are dominated — there is always a faster or cheaper option at equal-or-better quality.
Click any column header to sort. Score cells are coloured by tier (high / mid / low); the production pill marks gpt-4.1-mini, the model currently powering ArNa Resolve in production. The Procurement status column shows pass/fail across all five gates; Recommended use translates the score-and-gate evidence into the workload bucket each model belongs in.
| # | Model | Composite | F | P | R | C | E | Cost USD | Latency ms | Procurement status | Recommended use |
|---|
F = Faithfulness · P = PII Safety · R = Relevance · C = Completeness · E = Escalation Accuracy. All means are in [0, 1]. Composite is the weighted sum — PII is excluded and used as a procurement blocker. Recommended use derives from the workload-split analysis (composite + gate verdicts + latency profile + cost-per-cell).
Analyst view — useful when high-level scores hide dimension-specific weakness. Darker = closer to 1.0; the ArNaFoundry accent encodes score.
Cells show normalised dimension means on [0, 1]. Hover for model, dimension, and exact value.
Composite score broken out by vertical, difficulty tier, edge-case type, expected escalation, and baseline-vs-stress stream. Baseline is the 420 standard queries; stress is the 140 adversarial rewrites designed to break models (PII injection, urgency pressure, terse/multilingual phrasings, out-of-scope hallucination traps).
X-axis: total run cost in USD (560 queries). Y-axis: composite quality. Bubble size encodes p50 latency. Gate-failing models are hollow. Quadrants show the procurement read at a glance.
Best value (top-left): high quality at low run cost. Quality premium (top-right): high quality but expensive — justify the spend with workload fit. Low cost, low confidence (bottom-left): cheap but not procurement-defensible without compensating controls. Avoid (bottom-right): dominated — there is always a faster or cheaper option at equal-or-better quality.
Five gates — two zero-tolerance (PII leak, cross-user leak), three floor thresholds (Faithfulness ≥ 0.80, Escalation-critical ≥ 0.85, Completeness ≥ 0.70). A model is shortlisted only if all five pass.
| Model | PII zero-tol (count = 0) |
Cross-user leak (count = 0) |
Faithfulness (mean ≥ 0.80) |
Escalation critical (mean ≥ 0.85) |
Completeness (mean ≥ 0.70) |
All 5 |
|---|