Just take me to the code!
Pilot Helps To Answer These Questions
- How long shall I run this benchmark to get a precise results?
- Is my new algorithm really 3% faster than baseline or is that an error?
Pilot Is Designed To
- help testers who may not have enough statistics knowledge
- get accurate, precise, repeatable results
- get results using shortest possible time while still meet statistical requirements
Pilot Can Be Used By
- researchers, engineers, testers, users
a lightweight C++ library (BSD 3clause or GPLv2+ license) with C/C++ macro, library, and CLI interfaces
- can handle statistical stuff so your benchmark results are statistically valid
- measure and reduce the autocorrelation among samples
- calculate the confidence interval (CI)
- use t-distribution to calculate the required number of samples to achieve the desired CI
- can detect (and remove) warmup and cooldown phases using multiple methods
- can compare benchmarks that have very close results
- is optimized to get results using the shortest time (for both measurement and comparison)
- provides many easy-to-use interfaces
- is extensible through plugins
- and more …
Our MASCOTS’16 paper contains technical details of Pilot.
Join the mailing list to receive future release announcements or share your experiences:
This is a research project from the Storage Systems Research Center in UC Santa Cruz. This research was supported in part by the National Science Foundation under awards IIP-1266400, CCF-1219163, CNS-1018928, CNS-1528179, by the Department of Energy under award DE-FC02-10ER26017/DESC0005417, by a Symantec Graduate Fellowship, by a grant from Intel Corporation, and by industrial members of the Center for Research in Storage Systems.
This project does not reflect the opinion or endorsement of the sponsors listed above.