Better Thinking or a Bigger Model? Thinking–Answering Shuffles with Qwen3 on GPQA - Khalafyan, Edvard A. (Mathematical Problems of Computer Science, 2025)
We show that for Qwen3, large language models (LLMs) on the Graduate-Level Google-Proof Question Answering (GPQA) benchmark, thinker quality dominates answerer size: a 14B thinker paired with a 0.6B answerer reaches 54.24% accuracy, close to the 14B→14B diagonal (59.15%), whereas a 0.6B thinker reduces a 14B answerer to 20.54%. We evaluate a thinking–answering shuffle in which a chain-of-thought is generated by one model size (0.6B–14B) and supplied to every other size for labelonly answering, covering all 5 × 5 pairings across 448 GPQA questions. Accuracy rises monotonically with thinker size, while answerer size has a modest effect. Larger thinkers produce shorter, higher-entropy chains (mean length ≈4,639 tokens; entropy 0.416) than smaller thinkers (14,566; 0.404), and these properties correlate with better cross-model transfer. Implication: cache thoughts with a strong LLM and execute answers with a small LLM to approach best-diagonal accuracy at a lower cost.