ScottRBK has released eval-harness, a tool designed to generate personal evaluations for both LLMs and the agentic CLI harnesses they operate within. The project addresses the need to assess model performance in actual usage contexts rather than relying on general benchmarks or intuition.
- The repository allows users to build private lists of evaluations held away from public datasets.
- It includes example evaluations to demonstrate different patterns for assessing models and harnesses.
- A set of skills is provided to help CLI agents generate these evaluations, though the author notes they can be brittle.
- The tool aims to help users decide when to switch between local models like qwen3.6-27b and cloud models based on empirical data.
The framework helps users make informed decisions about model selection for professional workflows by providing a structured way to test capabilities in specific agentic environments.