Repeatability in Computer Science

Your `validation process' is, frankly, absurd and therefore provides useless results.

In our case
the version we posted had unnecessary dependency (file) in makefile, all you had to do is follow the error message and remove it and all would work fine- or ask us by mail
you did neither.

Besides, you only checked that projects compile; what exactly does it mean that the project compiles? to really check reproducibility you must validate it by running and comparing results and functionality. Your `test` would pass a an "hello, world!" program... absurd. You wasted more time for us in understanding your useless `evaluation` and writing such response as this, than you invested in evaluating our project. A waste of time.

We will make it publicly available soon, and the link will be http://www.carch.ac.cn/~huimin/main.html

We originally classified this paper as EM_no. We found that the authors had emailed us but we missed the email. We have changed the classification to EM_yes.

Please read carefully the build notes.

We originally classified this paper as Web and OK_leq_30. In the survey response, the authors said we only compiled part of the relevant code and that the remainder is not available. Therefore, we have changed the classification to EM_no.

Your study on repeatability in computer science research is quite interesting. Looking forward to see how it will impact the publication process and culture of this area of engineering.

Sharing code is unfortunately not possible due to corporate IP concerns.

We originally classified this paper as EM_no because did not receive code within our two month window after the initial request. In the survey response, the authors said that the code is now available. Our policy is to consider the code available at the time of the study. Therefore, we have kept the original classification of EM_no.

From the build notes, it seems that you had a trouble to set up the Postgres environment. Code that runs dependently on another system is unfortunately hard to get built. I am glad you eventually got it. But having a cross-platform code release is not very realistic for research publication. Thus, we have also published a virtual machine with everything built in.

A 30-minute verification that an author's code compiles, runs, and prints the expected output is interesting but is not a convincing test of scientific reproducibility.

A better test of reproducibility of a publication's results would be to independently implement and measure the algorithm or system described. This would be unlikely to replicate any bugs or systematic measurement errors, increasing the meaningfulness of a positive reproducibility result. Negative results would also be more likely to provide technical insight and foster scientific discussion.

Note that publishing code is also a rather thankless task. Likely outcomes include:
(a) Nobody will look at it and you will have wasted your time.
(b) Anyone who looks at it will see that it perfectly matches the paper's specification (except for being MUCH harder to read) and won't have learned anything.
(c) Someone will try to use it, run into some kind of trouble, and bother you with annoying questions.
(d) Someone will catch some trivial bug, approximation, or limitation and use it to cast doubt on your results.
(e) Some professor will convince a naive first-year student to use your code as a starting point for their experiments and you will have contributed to that poor student's misery and anguish.

We originally classified this paper as EM_no. We did not receive code within our two month window after the initial request. In the survey response, the authors said that the code is now available. Our policy is to consider the code available at the time of the study. Therefore, we have kept the original classification of EM_no.

Updates to the source code of MCJoin will be made available on this webpage:

http://homepage.cs.latrobe.edu.au /zhe/ZhenHePublications.html

In fact, our code is very easy to build. It is somewhat unfair to claim our code cannot be successfully built without asking for our assistance when you meet some problems during building our code.

Publishing or giving out code is easy. The hard part is to find enough resource to clean it up, make it portable, document it, and support it especially when the main goal of a project is not to build a widely used open source system but to demo ideas of a paper. Anyone with large system experience could imagine that there will be many system specfic hacks, shortcuts, and workarounds in the implementation of distributed system with many moving parts. For example, we may hard code a storage path or make some assumptions about access control configuration of the underlying systems. Such things will definitely not impact the validity of our results. And they can be smoothed out from the system given enough engineering resouce. Before that happen, we had to do substantial support work to help people deploy and run our system in the past several cases.

We originally classified this paper as NC. In the survey response, the authors said the paper is BC. Therefore, we have reclassified it as BC, downloaded the code and attempted to build it.

We originally classified this paper as EM_no, because we received an email that said the code was not available. In the survey response, the authors said the paper has published code. We have kept the classification as EM_no.

Our experimental setup needs several involved steps and not all of them are automated currently. It requires using several tools and systems that are not widely (or freely) available such as Wind River Simics, an UltraSPARC system for compiling and disassembling SPARC V9 binaries, and a cluster of machines to perform fault injection campaigns. This makes it complicated for us to package it for release

We would have liked to release the code publicly, and we have shared it with research groups have requested it. However, university policy makes public release difficult.

We originally classified this papers as NC. In the survey response, the authors said that the paper is BC. Therefore, we have reclassified it as BC, downloaded the code and built it.

The work is somewhat easy to reproduce with the amount of information given in the paper. Since the code has been used commercially, it is a lengthy procedure to make it open source.

In the first version of this study, we did not attempt kernel builds. In the final version, we did.

We originally classified this paper as EM_no. We did not receive code within our two month window after the initial request. In the survey response, the authors said that the code is now available. Our policy is to consider the code available at the time of the study. Therefore, we have kept the original classification of EM_no.

Interesting effort. However, I am not surprised that only a minority of projects can be successfully built/run by another group; that demands a pretty high quality bar of the code (simply managing build environments is often a full-time job for professional software development) and I suspect most academic projects do not warrant curating code to that level of quality, as the code is often not reused after the end of the project.

You didn't actually try to build it, but you gave it a pass! The comment about requiring a fresh installation of Ubuntu 12.10 was wrong, although this misunderstanding might have arisen from the instructions I emailed to you. An existing Ubuntu 12.10 installation was sufficient; more precisely, a system with a particular kernel version was required to build and run a kernel module. In any case, given that you sampled papers from systems conferences and didn't try to build a kernel module, I have doubts about the accuracy of your data and the soundness of your method. Then again, I only looked at my data point, so take my criticism with a grain of salt.

In the first version of this study, we did not attempt kernel builds. In the final version, we did.

This work has been extended to add backends for Fortran and X10. Current versions are available as part of the Mclab system at https://github.com/Sable/mclab.

We originally classified this paper as OK_gt_30. In the survey response, the authors said that we did not attempt to build all the relevant code. Therefore, we attempted to build the remaining code.

Large percentage of the work associated to this paper is theoretical. The details of the soundness proof and formalisation is available in the technical report. The code is part of a confidential project that has not been disclosed by Microsoft. I hope that that project will be open-sourced.

I do not like the way this study was conducted and found the follow-up work from Brown University more palatable, but I have not followed all the ripples from the initial "results" closely.

In the survey response, the authors said that the code we found was not correct and gave a link. The code we downloaded was from the same link that was given. Therefore, we have kept the original classification of code from Web.

We are happy to share the benchmark code and scripts with individual academics so that they can run and verify the code at their end.

Instructions on building and compiling Habanero-Scala is available at http://habanero-scala.rice.edu/. The link is also available on the paper.

In the survey response, the authors classify the paper as not having code available. We downloaded their code using a link from the paper. Therefore, we have kept the classification of code from Article.

As with the last time you folks inspected our software, you appear to be using an old build. Since April 18, 2014 (nearly 5 months ago), our software has used Scala's SBT for our build system. Your notes indicate that you tried (and failed) to use the old Makefile-based build system.

If you're having trouble building, why not correspond with us? This is not commercial software; we are academics with limited resources so our testing is similarly limited. Most of these issues can probably be ironed out with a brief conversation.

If you're interested in the original version of AutoMan that was the subject of the paper, you can sidestep your build problems. We distribute a binary JAR here, which is the link that you used: http://plasma.cs.umass.edu/emery/automan

In the survey response, the authors said that the version of the code we attempted to compile was outdated. We attempted to compile the version that was available at the time. Our policy is to consider code available at the time of the study. Therefore, we have kept the original classification.

The code, of the system and all of the example applications, is available and has been for the last year. It was not available at the time of publication at OOPSLA because we were exploring commercialization options and the University did not want us to release it.

We originally classified this paper as EM_no. In an email response the authors said that the code was not available due to possible commercialization. Then, in the survey response, the authors said that the code is now available. Our policy is to consider the code available at the time of the study. Therefore, we have kept the original classification of EM_no.

New versions of DistAlgo are now available also on sourceforge (https://sourceforge.net/projects/distalgo/) in case you need. Thank you for your work.

I think it's fantastic that you have undertaken this comprehensive study.

The static analyzer cccheck is publicly available on Windows and is widely used (see DevLabs > Code Contract forum). The source code of the static analyzer cccheck is proprietary.

A repeatability study takes more than clicking on links and running make scripts or the resulting executables. See for example Casey Klein et al at POPL 2012 for a repeatability study that went into depth. When we ran this study, we spent almost two years with one PhD student per case study.

-- Matthias Felleisen

We originally classified this paper as OK_lt_30. In the survey response, the authors said that we did not actually build the relevant code. Therefore, we attempted to build the relevant code.

updated codebase:
https://github.com/tech-srl/prime

Originally, we classified this paper as BC. We were given relevant code in an email exchange and we found an online implementation for the paper. In the survey, the authors say that the paper is NC. We have kept the original classification of BC.

We aimed in our paper to provide enough information to allow research groups to reproduce our results, i.e., to reimplement the concepts in our analysis. We believe that we accomplished this.

We made the code available as a courtesy on request, and were prepared to spend time and energy enabling interested researchers to replicate and extend our results, i.e., to use our code base. We did not aiming to provide a distribution of our code that works robustly out of the box. While we may aim to do this with future research projects, it was not an aim of the current work.

Our work is useful for researchers working with jikesrvm, btrace etc. Researchers need to understand how to build/use these tools before they look at our work. We strongly suggest that your study applies at least the same standards of evaluation as those used by the ECOOP Artifact Evaluation process.

The project continues to evolve on GitHub: https://github.com/solodon4/Mach7.

We also had a follow-up publication on the mentioned larger pattern-matching library at GPCE'13 (dl.acm.org/citation.cfm?doid=2517208.2517222) with Extended Abstract of it appearing in SPLASH'13 (dl.acm.org/citation.cfm?doid=2508075.2508098).

Updated and improved versions of this tool are available at https://github.com/philnguyen/soft-contract/ (full system) and http://dvanhorn.github.io/pcf/ (simple illustrative version).

The latest version (v0.5.2) at https://github.com/molecule-labs/molecule/ can be cross-compiled to scala 2.9.3 and 2.10. It confirms the benefits of an embedded DSL. With almost zero effort, we adapted the code to support developers leveraging the evolution of Scala and still running the code on JVM1.6 or above.
When using the scala2.10 compiler option to target Java7, the byte code will make use of invokedynamic. This will probably speed up our code even more. (I haven't yet re-run the same benchmarks on the same hardware to be able to measure the difference.)

Now the documentation explains how to run the examples.

In the survey response, the authors said that we did not find the correct code. We used a link that was given both in an original email and in the survey. The link was missing from the our record. Therefore, we have added the link and kept the original classification of EM_yes.

Thanks for doing a repeatability study, we should have more of them!

Very worthwhile study. Thanks for doing this. I know you caught a lot of flak for the initial results, but I really appreciate the followup and care that you've taken.

We originally classified this paper as Em_yes. In the survey response, one author said that we didn't find the correct code and the other two said that we did. The link that we used matches links given in the survey. We have kept the original classification of EM_yes.

We originally classified this paper as BC. In the survey response, the authors said that the paper is NC. The paper has a link to code for a proof checker. Therefore, we have kept the original classification of BC.

The build process failed because LLVM was not built from scratch. Please see the public release of the tool at hpcrl.cse.ohio-state.edu/wiki/index.php/DDGVec and the build instruction in file README