benchkit: A Declarative Framework for Composable Performance Evaluation of System Software

Performance-evaluation pipelines in systems research often combine benchmarks, system configuration steps, profiling tools, and analysis scripts. In practice, these components are glued together with ad-hoc shell scripts, notebooks, and bespoke tooling, making experiment dimensions difficult to explore systematically and results hard to reproduce or extend. We present benchkit, a lightweight Python library that provides a structured way to express performance experiments declaratively and to automate their full lifecycle—from build and execution to system configuration, profiling, and result collection. Instead of relying on monolithic scripts, benchkit provides a structured way to compose existing system tools (eg, CPU-placement utilities, frequency controllers, and performance profilers) while keeping benchmark code untouched.
We illustrate benchkit through two representative studies:(1) a drilldown of performance anomalies in SPEC CPU workloads on hybrid-core x86 processors, enabled by systematic exploration of CPU placement policies; and (2) an analysis of lock implementations and scheduling strategies on a many-core ARM server, where benchkit coordinates system tools and visualizations to interpret performance differences. We evaluate the overhead of benchkit and show that it introduces no measurable cost compared to handwritten shell workflows, both on the host and inside containers. These results show that benchkit provides a reproducible, extensible, and principled foundation for system-level performance experimentation.