The author proposes a set of reporting standards for Vision-Language-Action (VLA) attack papers to improve result transparency and comparability. The proposal argues that current literature often lacks essential baseline data and statistical context, leading to ambiguous claims about attack efficacy.
Key requirements include:
- Reporting a matched benign control rate at the same task and seed without subtraction.
- Using Wilson intervals for success rates over small sample sizes (e.g., n=50) instead of normal approximations.
- Explicitly stating whether the target was a stub, local checkpoint, or hosted model to clarify transfer claims.
- Including query budgets in results tables for search-based attacks to contextualize failure counts.
The author seeks community feedback on whether these standards are appropriate and asks if other subfields already employ similar reporting practices.