Rerun-ability, Repeatability, and Reproducibility in Experimentation

A panel of cyber security experimentation experts will discuss the concepts of rerun-ability, repeatability, and reproducibility in cyber security experimentation. Key questions to be discussed include: 1) What are the differences between the three R's and when is each important? 2) What counts as having repeated an experiment? 3) Given that it may be impossible to fully replicate an entire experiment, how does one determine which aspects of an experiment are critical for replication and which are not? 4) What should be done if the results of a reproduced experiment differ from that of the original?

Panelists

Roy Maxion is a research professor in the Computer Science and Machine Learning Departments at Carnegie Mellon University (CMU). He is also director of the CMU Dependable Systems Laboratory where the range of activities includes computer security, behavioral biometrics, insider detection, usability, and keystroke forensics as well as general issues of hardware/software reliability. In the interest of the integrity of experimental methodologies, Dr. Maxion teaches a course on Research Methods for Experimental Computer Science, and authored a 2012 paper entitled "Making Experiments Dependable". He is on the editorial boards of IEEE Security & Privacy and the International Journal of Biometrics, and is past editor of IEEE Transactions on Dependable and Secure Computing and IEEE Transactions on Information Forensics and Security. Dr. Maxion is a Fellow of the IEEE.

Terry Benzel is Deputy Director Cyber Networks and Cyber Security at the University of Southern California Information Sciences Institute (USC ISI). She is the technical project lead for the Cyber Defense Technology Experimental Research (DETER) testbed projects funded by DHS, NSF and DARPA. The projects are developing an experimental infrastructure network and scientifically rigorous testing frameworks and methodologies to support the development and demonstration of next-generation information security technologies for cyber defense.

Lance G. Joneckis is with The Institute for Defense Analyses, a federally funded research and development corporation providing objective analyses of national security issues. Dr. Joneckis recently completed a study to identify options that could enable major advances in cyber security capabilties. In addition to his work for the federal government, Dr. Joneckis has also worked in private industry in the areas of computer and communications. Dr Joneckis received a Ph.D. in Physics from The University of Maryland in 1990.