An analysis of coding agent tasks using GPT-5.6-sol and Codex CLI 0.144.1 demonstrates that reducing context tokens by 90% does not result in proportional cost savings for the complete task.
- A repository rewrite benchmark comparing no plugin, Ponytail, and isolated RTK plugins showed minimal cost differences despite token variance.
- Ponytail achieved 7.56% fewer tokens and 8.87% lower cost but required 13.51% longer duration compared to the baseline.
- RTK resulted in 13.20% more tokens and 7.18% higher cost with 44% more rounds than the baseline.
- Cached input accounted for 96.46% of tokens and 63.91% of cost, while model output was only 0.38% of tokens.
- High run-to-run variance (30-51%) indicates that small mean changes are insignificant compared to natural swings.
The author argues that token-saving tools should be evaluated on cost per verified completed task, including repeated-run variance, latency, and quality metrics.