DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems: plain AI paper summary
AI-generated summary. Read the original paper before relying on it.
Hugging Face paper page · Original paper on arXiv
DecisionPivot tests whether decision models stay correct when a task changes in a controlled way, rather than trusting one aggregate accuracy number. Its contrast groups, confidence tests, and selective-coverage measures expose failures hidden by ordinary scores.
Key findings
- The benchmark has 48 tasks and 40,033 items across seven decision roles.
- Its core track contains 7,800 solver-verified items from 13 generated task families.
- Two 27B models separated by 2.6 reference-track points reached 70.1% and 44.2% contrast-group accuracy.
- Perplexity Decider ranked positive examples above negative ones in 92.3% of pairs but answered both correctly in only 74.1%.
How you might use it
- Evaluate an agent router on paired requests that differ in one important detail, then compare its confidence ordering with whether it made both decisions correctly.
Limits
The abstract does not report how well these benchmark rankings predict failures in deployed customer workflows.
Related research
- StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions This paper on arXiv Related paper on arXiv: Both papers evaluate confidence-aware decisions. DecisionPivot measures contrast accuracy and calibration, while StartLux-Decision trains models to reason selectively when confidence is low.
Business ideas
1. decision contrast suite
Offer: A delivered decision contrast suite for contrast-group decision testing.
Buyer: LLM evaluation teams
Technical link: Use DecisionPivot contrast groups, verified solvers, and multiple decision roles.
Evidence
- competitor: LangSmith integrates with pytest, Vitest, and GitHub workflows for PR or nightly evaluations. (scope: vendor documentation year: 2026 region: global) (source)
- adoption: The proportion of UK businesses with 10 or more employees reporting use of at least one AI technology rose from around 12% to around 35%. (scope: ONS Business Insights and Conditions Survey analysis year: 2026 region: United Kingdom denominator: businesses with 10 or more employees) (source)
Differentiation: Arize Phoenix supports evaluation and experimentation; add solver-backed contrast diagnostics.
First test: Run a paper-specific trial 1 on 100 labeled customer examples and record baseline errors and reviewer corrections.
Success measure: Proposed acceptance for outcome 1: at least 85% of held-out examples meet the customer quality threshold.
Constraints: Pilot only. Use customer-owned or licensed data, approved runtimes, and human review. Market evidence is contextual proxy evidence rather than TAM.
2. confidence calibration report
Offer: A delivered confidence calibration report for contrast-group decision testing.
Buyer: LLM evaluation teams
Technical link: Use DecisionPivot contrast groups, verified solvers, and multiple decision roles.
Evidence
- competitor: Arize Phoenix is open-source and local-first for tracing, evaluation, experimentation, and prompt iteration. (scope: vendor product description year: 2026 region: global) (source)
- adoption: The proportion of UK businesses with 10 or more employees reporting use of at least one AI technology rose from around 12% to around 35%. (scope: ONS Business Insights and Conditions Survey analysis year: 2026 region: United Kingdom denominator: businesses with 10 or more employees) (source)
Differentiation: Arize Phoenix supports evaluation and experimentation; add solver-backed contrast diagnostics.
First test: Run a paper-specific trial 2 on 100 labeled customer examples and record baseline errors and reviewer corrections.
Success measure: Proposed acceptance for outcome 2: at least 85% of held-out examples meet the customer quality threshold.
Constraints: Pilot only. Use customer-owned or licensed data, approved runtimes, and human review. Market evidence is contextual proxy evidence rather than TAM.
3. guardrail regression gate
Offer: A delivered guardrail regression gate for contrast-group decision testing.
Buyer: LLM evaluation teams
Technical link: Use DecisionPivot contrast groups, verified solvers, and multiple decision roles.
Evidence
- competitor: LangSmith integrates with pytest, Vitest, and GitHub workflows for PR or nightly evaluations. (scope: vendor documentation year: 2026 region: global) (source)
- adoption: The proportion of UK businesses with 10 or more employees reporting use of at least one AI technology rose from around 12% to around 35%. (scope: ONS Business Insights and Conditions Survey analysis year: 2026 region: United Kingdom denominator: businesses with 10 or more employees) (source)
Differentiation: Arize Phoenix supports evaluation and experimentation; add solver-backed contrast diagnostics.
First test: Run a paper-specific trial 3 on 100 labeled customer examples and record baseline errors and reviewer corrections.
Success measure: Proposed acceptance for outcome 3: at least 85% of held-out examples meet the customer quality threshold.
Constraints: Pilot only. Use customer-owned or licensed data, approved runtimes, and human review. Market evidence is contextual proxy evidence rather than TAM.
4. solver-backed eval service
Offer: A delivered solver-backed eval service for contrast-group decision testing.
Buyer: LLM evaluation teams
Technical link: Use DecisionPivot contrast groups, verified solvers, and multiple decision roles.
Evidence
- competitor: Arize Phoenix is open-source and local-first for tracing, evaluation, experimentation, and prompt iteration. (scope: vendor product description year: 2026 region: global) (source)
- adoption: The proportion of UK businesses with 10 or more employees reporting use of at least one AI technology rose from around 12% to around 35%. (scope: ONS Business Insights and Conditions Survey analysis year: 2026 region: United Kingdom denominator: businesses with 10 or more employees) (source)
Differentiation: Arize Phoenix supports evaluation and experimentation; add solver-backed contrast diagnostics.
First test: Run a paper-specific trial 4 on 100 labeled customer examples and record baseline errors and reviewer corrections.
Success measure: Proposed acceptance for outcome 4: at least 85% of held-out examples meet the customer quality threshold.
Constraints: Pilot only. Use customer-owned or licensed data, approved runtimes, and human review. Market evidence is contextual proxy evidence rather than TAM.
5. decision model scorecard
Offer: A delivered decision model scorecard for contrast-group decision testing.
Buyer: LLM evaluation teams
Technical link: Use DecisionPivot contrast groups, verified solvers, and multiple decision roles.
Evidence
- competitor: LangSmith integrates with pytest, Vitest, and GitHub workflows for PR or nightly evaluations. (scope: vendor documentation year: 2026 region: global) (source)
- adoption: The proportion of UK businesses with 10 or more employees reporting use of at least one AI technology rose from around 12% to around 35%. (scope: ONS Business Insights and Conditions Survey analysis year: 2026 region: United Kingdom denominator: businesses with 10 or more employees) (source)
Differentiation: Arize Phoenix supports evaluation and experimentation; add solver-backed contrast diagnostics.
First test: Run a paper-specific trial 5 on 100 labeled customer examples and record baseline errors and reviewer corrections.
Success measure: Proposed acceptance for outcome 5: at least 85% of held-out examples meet the customer quality threshold.
Constraints: Pilot only. Use customer-owned or licensed data, approved runtimes, and human review. Market evidence is contextual proxy evidence rather than TAM.