Pilot Benchmark Framework

Just take me to the code!

Pilot Helps To Answer These Questions

  • How long shall I run this benchmark to get a precise results?
  • Is my new algorithm really 3% faster than baseline or is that an error?

Pilot Is Designed To

  • help testers who may not have enough statistics knowledge
  • get accurate, precise, repeatable results
  • get results using shortest possible time while still meet statistical requirements

Pilot Can Be Used By

  • researchers, engineers, testers, users

Pilot Is

a lightweight C++ library (BSD 3­clause or GPLv2+ license) with C/C++ macro, library, and CLI interfaces

Pilot’s Function

  • can handle statistical stuff so your benchmark results are statistically valid
    • measure and reduce the autocorrelation among samples
    • calculate the confidence interval (CI)
    • use t­-distribution to calculate the required number of samples to achieve the desired CI
  • can detect (and remove) warm­up and cool­down phases using multiple methods
  • can compare benchmarks that have very close results
  • is optimized to get results using the shortest time (for both measurement and comparison)
  • provides many easy­-to­-use interfaces
  • is extensible through plug­ins
  • and more …

Publication

Our MASCOTS’16 paper contains technical details of Pilot.

Mailing lists

Join the mailing list to receive future release announcements or share your experiences:

If you don’t mind spending time testing bleeding edge code and giving us feedback in the Pilot alpha test, please let me (yanli AT ascar.io, or Twitter) know and join us on Slack.

Acknowledgments

This is a research project from the Storage Systems Research Center in UC Santa Cruz. This research was supported in part by the National Science Foundation under awards IIP-1266400, CCF-1219163, CNS-1018928, CNS-1528179, by the Department of Energy under award DE-FC02-10ER26017/DESC0005417, by a Symantec Graduate Fellowship, by a grant from Intel Corporation, and by industrial members of the Center for Research in Storage Systems.

This project does not reflect the opinion or endorsement of the sponsors listed above.