Artifact Evaluation Process
Authors of accepted PACT 2021 papers are invited to formally submit their supporting materials for Artifact Evaluation. Submission is voluntary. The Artifact Evaluation process is run by a separate committee whose task is to assess if artifacts supporting the work described in the papers are available and functional, and (some of) experimental results are reproducible.
We encourage authors prepare their artifacts for submission and make them more portable, reusable and customizable using open-source frameworks including OCCAM, Docker, reprozip, CodeOcean and CK. For example, CGO’17 paper with CK-based experimental workflow and interactive dashboard won distinguished artifact award.
Papers that successfully go through the Artifact Evaluation process will receive a seal of approval printed on the papers themselves. Authors of such papers will have an option to include their Artifact Appendix to the final paper (up to 2 pages). Authors are also encouraged (though not obliged) to make their artifacts publicly available upon publication of the proceedings, for example using Code Ocean platform (official partner of IEEE Digital Library).
Artifact Evaluation Deadlines
Artifact Submission: July 26, 2021
How to submit
You should prepare your artifacts for submission using the following guide. Then, register your submission at the HotCRP – you will be asked to submit your paper title, author list, artifact abstract, PDF of your paper with an appendix describing how to access and validate your artifacts, and describe possible conflicts of interests.
Reviewing process
Your artifacts will be reviewed according to the following guidelines. Artifacts receiving “met expectations” or above score will pass evaluation and will receive a stamp of approval.
Feedback
We consider Artifact Evaluation as a continuous learning curve – our eventual goal is to develop a common methodology for experiment sharing and evaluation in computer system’s research. Therefore, if you have questions, comments and suggestions on how to improve artifact submission, reviewing, customization and reuse, please do not hesitate to get in touch with the AE steering committee!
For additional information regarding your artifact evaluation submission, please contact the Artifact Evaluation Chairs, Bingsheng He (National University of Singapore) and Jidong Zhai (Tsinghua University).
Artifact Evaluation Committee Co-chairs:
Bingsheng He, National University of Singapore
Jidong Zhai, Tsinghua University
Artifact Evaluation Committee Co-chairs:
Marcos Horro, University of A Coruna
Utpal Bora, IIT Hyderabad
Shashin Halalingaiah, IIT Madras
Muhammad Aditya Sasongko, Koc University, Istanbul
Geonhwa Jeong, Georgia Tech
Yusuke Izawa, Tokyo Institute of Technology
Peihua Zhang, ICT, CAS, China
Kezhao Huang, Tsinghua University
Sixu Hu, National University of Singapore