LLM-Based task classifier tend to misroute prompts that look simple at first glance, but require deeper understanding - I call it "Type II Error" here.

Setup

TaskClassBench, a custom benchmark of 200 effective trap prompts (context-contradiction + disguised-correction categories) designed to create a mismatch between surface simplicity and contextual complexity.

For example:
System context establishes a fault-tolerant ETL pipeline with retry logic, dead-letter queues, and alerting. User message: "we don't need the retry logic actually." Four-word sentence, but it's an architectural revision with cascading implications. 8 Step-0 variants tested across 4 commercial models (DeepSeek, Gemini Flash, Claude Haiku, Claude Sonnet), temperature 0, 4 independent API rounds.

Key findings:

• Open-ended exploration "What's really going on here?" reduces Type II rate to 1.25% vs. 3.12% for directed extraction "Summarize the user's intent in one sentence"
• A content-free metacognitive directive ("Think carefully about the complexity of this task") achieves 1.0% - not significantly different from exploration - but I hypothesize it may differ under filled context (eg. 200k tokens in 1m window)
• Both significantly outperform structured detection "Are depth signals present? yes/no" and directed extraction
• Structured yes/no detection catastrophically harms Claude models: Haiku errors jump from 10 to 43 out of 200 (330% increase), Sonnet from 12 to 34 (183%)
• The mechanism appears to be forced attention to task complexity before classification, not open-ended framing specifically (which I still have high hopes for :D). What seems to matter is unbounded engagement. Structured approaches fail because they constrain or foreclose complexity signals.

The most unexpected finding

What I call "recognition without commitment": Claude Sonnet under "think carefully" writes "This request asks me to violate an established change management policy" in its Step-0 reasoning and still classifies Quick. Under exploration, the same model identifies the same violation and correctly escalates. The think-carefully instruction lets the model observe depth without committing to it; exploration forces a committed implication statement that anchors classification. This pattern is consistent across all 5 cases where exploration rescues think-carefully failures.

Effect is capability-moderated (I suppose)

DeepSeek and Claude Haiku drive the pooled result. Gemini Flash is near-ceiling at baseline (3/200 errors). Claude Sonnet shows a mixed 3:2 discordant pattern. The weaker the model, the larger the benefit. I hypothesise this relationship reverses at >100K context loads, where even capable models would need the scaffold but this is untested and stated as a falsifiable prediction.

Key limitations I want to be upfront about:

• Post-hoc expansion: Benchmark was expanded after R2 yielded p = 0.065 at N=120. The categories expanded (CC and DC) were chosen based on R1/R2 discrimination patterns, not blindly. All claims are exploratory, not confirmatory.
• Circularity risk: Ground truth labels were generated by Claude Sonnet 4.6 - one of the four models subsequently tested. Partially mitigated by 93.3% human agreement on N=30 subset, but the 160 expanded prompts have zero interrater validation.
• Heterogeneous effect: Pooled result is driven by 2 of 4 models. Gemini Flash near-ceiling, Sonnet mixed. The claim is better scoped as "helps models with moderate baseline error rates."
• Narrow scope: All prompts are short (<512 tokens). Proprietary models only. Single API run for the primary dataset.
• Cross-dataset ablation: R3 mechanism ablation is a separate API run, not within-run. The expl2 vs. think equivalence (p = 0.77) could be affected by run-to-run variance (bounded at +-2 errors, but still).
• Single author: I designed, built, labelled, and analysed everything. No independent replication.
• The paper has 18 explicitly stated limitations in total - I'd be glad to receive your opinions and possibly hints :).

Links

• Paper (32 pages with full appendices, all data table)
• Benchmark and experimental data

What I'm looking for

1. Interrater validation: If anyone is willing to label any number of trap prompts as Quick vs. requires-deeper-processing (binary or with categories), this would directly address the biggest methodological weakness. The prompts and contexts are in the repo.
2. Methodological critique: What did I miss? What would you do differently?
3. Replication on open-weight models: All my data is on commercial APIs. Would love to see if the pattern holds on Llama, Kimi, Qwen etc.
4. ArXiv endorsement: I'm an independent researcher without academic affiliation. If anyone with cs.CL or cs.AI endorsement privileges finds the work credible enough, I'd appreciate help getting it on arXiv.