Parallel's PWS-Decisions Benchmark Exposes OpenAI Falling Below Random Accuracy

A test for facts in motion

PWS-Decisions contains 300 closed-ended questions modeled on production tasks. Each item depends on information that is current, specialized or distributed across multiple web pages.

• Qualifying accounts from hiring, funding or company signals
• Checking current product capabilities
• Confirming security, compliance and policy claims
• Tracking events whose status changes quickly

The company says each answer was verified through separate web research and a direct source. A gated public split contains 100 questions on Hugging Face. Parallel keeps the other 200 private to reduce the risk that benchmark answers enter training or tuning data.

Static scores miss retrieval

JevBench evaluates typed decisions with the relevant evidence already included in the prompt. PWS-Decisions requires the system to locate evidence on the open web, judge its recency and authority, and select the corresponding answer.

OpenAI Decisions scored 90.0% on JevBench and 41.7% on PWS-Decisions. Jev scored 88.3% and 33.4%, respectively. Those declines show how strongly retrieval and freshness affect an otherwise similar classification task.

Parallel calculates expected accuracy from uniform random selection at 42.42%, accounting for the available choices across the dataset. OpenAI’s result falls slightly below that baseline, while Jev trails it by almost nine percentage points. Aggregate accuracy cannot separate stale retrieval, poor source selection, refusals and final-label errors, so category-level results would be needed to diagnose each system’s failures.

Search reshapes cost and latency

Web access adds search fees and round trips to a classification pipeline. Parallel compared its bundled Decisions API with OpenAI Luna and Anthropic Haiku 5.5 using native web search, then normalized cost to 1,000 questions.

In latency reporting, p50 is the median response time, while p95 is the threshold within which 95% of requests finish. Under Parallel’s test settings, the competitors’ median latencies were 11 to 13 times Parallel’s, and their p95 latencies were 15 to 27 times as high. Their normalized costs were about 17 to 29 times as high.

Prompt length, region, concurrency, caching, search configuration and current vendor prices can shift those figures. Production evaluations should replay representative questions under expected traffic and record both end-to-end latency and invoice-level cost.

Run the public split

The public package includes 100 multiple-choice questions, ground-truth labels, deterministic scoring and a minimal Parallel runner. Adapters for other models can use the same evaluation sequence:

1. Request access to the gated dataset.
2. Send each question and its allowed choices to the system under test.
3. Write one JSON Lines record for every dataset item.
4. Run the supplied scorer against the completed output file.

Each output line uses a fixed envelope and preserves the dataset ID:

{"id":"PWS-D-001","prediction":"<choice_value>","status":"success"}

Valid status values are success, refusal, error and invalid_response. Refusals, errors, malformed responses and missing predictions count as incorrect. Keeping every item in the denominator prevents abstention from inflating accuracy, while supplied labels allow deterministic scoring without an LLM judge.

Limits behind the headline

• Vendor ownership: Parallel designed the benchmark, controls the private labels and reports the leaderboard. Independent verification of the full ranking requires access or a third-party audit.
• Limited public coverage: External evaluators can reproduce scores on 100 questions, one-third of the full set. Results may shift with category mix and sample size.
• Changing ground truth: Web facts expire. Reliable longitudinal comparisons require versioned labels, source dates and a documented refresh process.
• Narrow output format: Multiple-choice accuracy measures label selection. It leaves citation quality, confidence calibration, open-ended synthesis and evidence completeness outside the score.
• Average-only reporting: Three-run averages can hide run-to-run variation. Deployment tests should retain individual results, failure statuses and category-level errors.

Built for pipeline decisions

Parallel positions Decisions for high-volume workflows that need structured classifications from current web evidence. Representative uses include:

• Matching CRM records as the same company, a subsidiary or unrelated entities
• Validating vendor security and compliance claims
• Qualifying leads from hiring, funding and expansion signals
• Checking supplier and press-release claims against primary sources
• Moderating new marketplace listings before publication

For compliance, moderation and other consequential workflows, teams should store the question, allowed choices, evidence date, system version and final status. Human review thresholds and periodic rechecks can reduce the impact of ambiguous or outdated evidence.

Product materials say one API call can contain up to 100 questions, and the free playground includes 5,000 requests. The benchmark chart expresses cost per 1,000 questions, so teams planning to batch work should confirm how current billing maps API calls, questions and billable requests.

PWS-Decisions gives developers a concrete starting point for evaluating live-web classification. A production comparison should add private domain questions, source audits, concurrency tests and current pricing from each vendor.