A Suite of Tools for Conducting Benchmarks

Benchmarking is a tedious task, which few enjoy, but every programmer or systems software researcher or developer must do. Benchmarking has little scientific value on its own, but it is needed to support other work. This project develops Auto-pilot, a suite of tools for producing accurate, reliable and informative benchmark results. There is an abundance of standard performance metrics available. Auto-pilot does not implement specific tests, but rather provides an infrastructure for running tests.

Auto-pilot includes a simple language to control benchmark execution; extensible file system benchmarking scripts as an example of how to use Auto-pilot; automated and modular analysis tools called Getstats; and automated graphing tool called Graphit that uses Gnuplot as a backend; and a 68-page user manual. Auto-pilot's configuration language includes support for checkpoints, which allow you to reboot and resume the machine between benchmark iterations without any manual intervention. Getstats produces tabular reports, CSV files for outside analysis, and can compare two samples using hypothesis testing. Using linear regression, Getstats can automatically find memory leaks or other problems that degrade performance over time.

We have successfully used Auto-pilot for over three years on twenty different projects, and it has saved us countless hours and days of work.

Mailing Lists:

Click on the list name below to access the subscription Web page. That page will also tell you how to post to the list, access the lists' archives, change your subscription status, etc.

List Name Purpose Who Can Subscribe Who Can Post
auto-pilot For users' and developers' questions, announcements, bug reports, etc. Anyone Subscribers only

Software:

Download software.

Conference and Workshop Papers:

# Title (click for html version) Formats Published In Date Comments
1 Auto-pilot: A Platform for System Software Benchmarking PS PDF BibTeX Usenix Technical Conference, FREENIX Track Apr 2005  

Manuals:

# Title (click for html version) Formats Published In Date Comments
1 The Auto-pilot Benchmarking Suite User Manual PS PDF BibTeX   Jul 2005 See an all-in-one version.

Past Students:

# Name (click for home page) Program Period Current Location
1 Charles P. Wright PhD May 2003 - May 2006 Research Staff Member, Network Server Systems Software group, IBM T. J. Watson Research Center (Hawthorne, NY)
2 Aditya Kashyap MS Sep 2003 - Dec 2004 Senior Analyst, Fixed Income Research group, Lehman Brothers (New York, NY)
3 Devaki Kulkarni MS Sep 2003 - Dec 2004 Senior Member of the Technical Staff, Performance and Storage groups, VMware, Inc. (Palo Alto, CA)
4 Miretskiy "Eugene" Yeugeniy MS May 2003 - Dec 2004 Software Engineer Google (New York, NY)
5 Jeffrey R. Osborn BS/MS Jan 2001 - May 2003 Senior Analyst, Fixed Income Research group, Lehman Brothers (New York, NY)
6 Charles P. Wright BS (Honors) Dec 2001 - May 2003 Research Staff Member, Network Server Systems Software group, IBM T. J. Watson Research Center (Hawthorne, NY)

Sponsors:

# Sponsor Amount Period Type Title (click for award abstract)
1 NSF Trusted Computing (TC) $400,000 2003-2006 Sole PI A Layered Approach to Securing Network File Systems
2 NSF CAREER (NGS) $400,000 2002-2007 Sole PI CAREER: An In-Kernel Runtime Execution Environment for User-Level Programs
3 HP/Intel $131,529 2002-2003 Sole PI Linux Application Performance and File System Security
4 HP/Intel $22,490 2001-2002 Sole PI Linux Network Scalability and File System Reliability

(Last updated: Tue Feb 6 21:04:26 EST 2007)