Authors of accepted research papers are invited to submit an artifact to the RE'19 Artifact Track. Research papers with “artifacts” receive a “badge” on the front page of their paper in the proceedings. These badges certify that the paper has associated artifacts of the following form:

Papers with such badges contain reusable products that other researchers can use to bootstrap their own research. Such papers earn increased citations and greater prestige in the research community. Artifacts of interest include (but are not limited to) the following. This list is not exhaustive, but if your proposed artifact is not on this list, please email the chairs before submitting.

• Software, which are implementations of systems or algorithms potentially useful in other studies. For instance, tools to search for optimally consistent requirements specifications.
• Machine readable requirements models, e.g. in IstarML or UML interchange formats.
• Traceability relations between artifacts, such as requirements to source code.
• Data repositories, which are data (e.g., requirements models, requirements text, survey raw data) that can be used for multiple software engineering approaches.
• Requirements in natural language form, e.g. textual requirements in a spreadsheet, DOORS, JIRA export.
• User reviews, such as app reviews, product changelogs, and release notes.
• Frameworks, which are tools and services illustrating new approaches to requirements engineering that could be used by other researchers in different contexts. For example, a service to highlight inconsistencies in natural language.

Selection Criteria

The RE19 artifacts track will be evaluated using the criteria in the last row of the above table. The goal of this track is to encourage reusable research products.
-> Hence, no “functional” badges will be awarded
Note that for “replicated” and “reproduced” badges, authors will need to offer appropriate documentation that their artifacts have reached that stage. So it is anticipated that most of our artifacts will be “reusable” and “available”.

Review Process

• Artifacts will be reviewed via a single blinded GitHub-based process. All reviews will be Github issues.
• Artifacts will be checked for their “badge worthiness” by one reviewer unless there is an author/reviewer dispute in which case a second check will be conducted by a second reviewer or one of the chairs.
• Reviewers will be assigned an anonymous GitHub ID.
• And authors will interact with the reviewers using their real (non-anonymously).
• Reviewers will have 7 days to assess the artifacts.
• Reviewers and authors will then have 3 days to interact such that (e.g.) if there is one line missing in a config file, then that bug can be fixed and the artifact declared “reusable”.
• Track chairs will then meet to write a “decisions.md” list written to the RE'19 Github that contains a table of submissions and their associated badge (if any). These will be stored as labels in the Github repo.

Submission Process

(Optional) Step1: Read “Good Enough Practices in Scientific Computing” by Greg Wilson, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, Tracy K. Teal. and consider if the advice of Section4 is useful for you. Also, read Section7 for a list of packaging options that, might be, overly elaborate (but you would need to decide that for yourself).

Step2: Prepare all the supporting materials needed to evaluate an artifact:

• a README.md main markdown file describing what the artifact does;
• A STATUS.md markdown file stating what kind of badge you are applying for (one of reusable, available, replicated, reproduced) as well as the reasons why you think your artifact deserves that badge.
• a LICENSE.md markdown file describing the distribution rights (and note that to score “available” or higher, then that license needs to be some form of open source license).
  • an INSTALL.md markdown file with installation instructions. These instructions should include notes on what output to expect that confirms
  • The code is installed and working
  • The code is doing something interesting and useful
    
• For artifacts where the code/ virtual machine/ data sets/ log files are too big to fit into a standard Github, repo, the above INSTALL.md should include the appropriate download instructions.
• Enough associated code and data such that some CS person with a reasonable knowledge of scripting, build tools, etc. could install, build, run your code; i.e. anyone that gets the joke above.

Step3: Place in a directory with some ID that refers to a contact author, plus a tool name, plus an RE 19 paper title/id number.

Step4: Zip that file, Place that file online.

Step5: Go to link and fill in a submission form which will ask you to include a link to your zip file.

Step6: If, in a few days, your directory has not appeared at RE'19 Github, then ping Neil Ernst or Tim Menzies

Key Dates

All deadlines are 23:59 Anywhere on Earth (Standard Time).

Artifact Track Co-Chairs

Any inquiries regarding tutorials can be directed to the Artifact Co-Chairs:

Artifact Evaluation Co-Chair

Neil Ernst

University of Victoria, Canada

Artifact Evaluation Co-Chair

Tim Menzies

North Carolina State University, USA

Artifact Track Committee

• Jennifer Horkoff, Chalmers/Gothenburg University, Sweden
• Kelly Blincoe, University of Auckland, New Zealand
• Jin Guo, McGill University, Canada
• Stefan Wagner, University of Stuttgart, Germany
• Daniel Méndez Fernández, Technical University of Munich, Germany
• Eric Knauss, Chalmers/Gothenburg University, Sweden
• Tanmay Bhowmik, Mississippi State University, USA
• Lloyd Montgomery, University of Hamburg, Germany