[hpc-announce] [Call for Papers] The First International Workshop on Evaluatology (Evaluatology 2024)

[gaowanling at ict.ac.cn]

[Apologies if you receive multiple copies of this CFP]

CALL FOR PAPERS

==========================================================

The First International Workshop on Evaluatology (Evaluatology 2024)

https://urldefense.us/v3/__https://www.benchcouncil.org/eva24/index.html__;!!G_uCfscf7eWS!eTSmofhZilgqjCiPiPdBcDpWioOqfxfbbe2oygz7D97nzyQqOAjVCM8rlj_19_FX89RB3jWU-SbmOqpBPwmsjhFXCphJ$ 

In conjunction with
the 16th BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench 2024)
https://urldefense.us/v3/__https://www.benchcouncil.org/bench2024/__;!!G_uCfscf7eWS!eTSmofhZilgqjCiPiPdBcDpWioOqfxfbbe2oygz7D97nzyQqOAjVCM8rlj_19_FX89RB3jWU-SbmOqpBPwmsjk-IsSkH$ 

* Dec. 4-6, 2024, Guangzhou, China

Important Dates

1-page abstract due date: October 15, 2024, AoE
Notification date: November 1, 2024, AoE
Workshop: December 4, 2024
Extended article due date: January 1, 2025, AoE

Submission Site: https://urldefense.us/v3/__https://eva2024.hotcrp.com/__;!!G_uCfscf7eWS!eTSmofhZilgqjCiPiPdBcDpWioOqfxfbbe2oygz7D97nzyQqOAjVCM8rlj_19_FX89RB3jWU-SbmOqpBPwmsjk6KAvYI$ 

==========================================================


Introduction
---------------------

Evaluatology 2024 is an international workshop dedicated to advancing the science and engineering of evaluation.

We cordially invite researchers from various fields to submit their work, with a strong emphasis on interdisciplinary research. Whether your research focuses on computers, artificial intelligence (AI), medicine, education, finance, business, psychology, or other social sciences, all relevant contributions are highly appreciated and encouraged.

Evaluatology 2024 follows a post-publication format, where we request abstracts of up to one page only. Following the presentation and feedback from the audience, the full article will be published in the Bench 2024 proceedings or the Tbench journal. To ensure a clear and concise focus on evaluation issues, please refer to "A Short Summary of Evaluatology: The Science and Engineering of Evaluation, J. F. Zhan, BenchCouncil Transactions on Benchmarks, Standards, and Evaluations."
( https://urldefense.us/v3/__https://www.sciencedirect.com/science/article/pii/S2772485924000279__;!!G_uCfscf7eWS!eTSmofhZilgqjCiPiPdBcDpWioOqfxfbbe2oygz7D97nzyQqOAjVCM8rlj_19_FX89RB3jWU-SbmOqpBPwmsjpzfqIRs$  )


Call for Papers
---------------------

** Evaluation theory and methodology
 - Formal specification of evaluation requirements
 - Development of evaluation models
 - Design and implementation of evaluation systems
 - Analysis of evaluation risk
 - Cost modeling for evaluations
 - Accuracy modeling for evaluations
 - Evaluation traceability
 - Identification and establishment of evaluation conditions
 - Equivalent evaluation conditions
 - Design of experiments
 - Statistical analysis techniques for evaluations
 - Methodologies and techniques for eliminating confounding factors in evaluations
 - Analytical modeling techniques and validation of models
 - Simulation and emulation-based modeling techniques and validation of models
 - Development of methodologies, metrics, abstractions, and algorithms specifically tailored for evaluations

** The engineering of evaluation
 - Benchmark design and implementation
 - Benchmark traceability
 - Establishing the least equivalent evaluation conditions
 - Index design, implementation
 - Scale design, implementation
 - Evaluation standard design and implementations
 - Evaluation and benchmark practice
 - Tools for evaluations
 - Real-world evaluation systems
 - Testbed

** Data set
 - Explicit or implicit problem definition deduced from the data set
 - Detailed descriptions of research or industry datasets, including the methods used to collect the data and technical analyses supporting the quality of the measurements
 - Analyses or meta-analyses of existing data
 - Systems, technologies, and techniques that advance data sharing and reuse to support reproducible research
 - Tools that generate large-scale data while preserving their original characteristics
 - Evaluating the rigor and quality of the experiments used to generate the data and the completeness of the data description

** Benchmarking
 - Summary and review of state-of-the-art and state-of-the-practice
 - Searching and summarizing industry best practice
 - Evaluation and optimization of industry practice
 - Retrospective of industry practice
 - Characterizing and optimizing real-world applications and systems
 - Evaluations of state-of-the-art solutions in the real-world setting

** Measurement and testing
 - Workload characterization
 - Instrumentation, sampling, tracing, and profiling of large-scale, real-world applications and systems
 - Collection and analysis of measurement and testing data that yield new insights
 - Measurement and testing-based modeling (e.g., workloads, scaling behavior, and assessment of performance bottlenecks)
 - Methods and tools to monitor and visualize measurement and testing data
 - Systems and algorithms that build on measurement and testing-based findings
 - Reappraisal of previous empirical measurements and measurement-based conclusions
 - Reappraisal of previous empirical testing and testing-based conclusions


Paper Submission
---------------------

Papers must be submitted in PDF. The review process follows a strict dual-anonymous policy per the established Bench conference norms. The submissions will be judged based on the merit of the ideas rather than the length. After the conference, the proceedings will be published by Springer LNCS or TBench journal (Indexed by EI).
At least one author must pre-register for the symposium, and at least one author must attend the symposium to present the paper. Papers for which no author is pre-registered will be removed from the proceedings.
 - The submission must describe novel, unpublished work that is not currently under review of any other conference venue or journal.
 - All authors and affiliation information must be anonymized.
 - Paper must be submitted in printable PDF format (LNCS format).
 - Please number the pages of your submission.
 - The submission must be formatted for black-and-white printers.
 - Please make sure your figures are readable when printed in black and white.
 - LNCS LaTeX template: https://urldefense.us/v3/__https://www.benchcouncil.org/file/llncs2e.zip__;!!G_uCfscf7eWS!eTSmofhZilgqjCiPiPdBcDpWioOqfxfbbe2oygz7D97nzyQqOAjVCM8rlj_19_FX89RB3jWU-SbmOqpBPwmsjjmSlXVO$ 


Program Committees
---------------------

Program Chairs
Jianfeng Zhan, ICT, Chinese Academy of Sciences, China
Weiping Li, Civil Aviation Flight University of China

Program Associate Chairs
Maodeng Li, Beijing institute of control engineering, China
Wanling Gao, ICT, Chinese Academy of Sciences, China

Technical Program Committees
Hajdi Cenan, AIrt, Croatia
Wei Wang, East China Normal University, China
Jialiang Tan, Lehigh University, USA
Dong Li, Zhongguancun Lab, China
Tong Wu, National Institute of Metrology, China
Davor Runje, AIrt, Croatia
Jianming Tang, Hangzhou Dianzi University, China
Lei Wang, ICT, Chinese Academy of Sciences, China
Xingzhou Zhang, ICT, Chinese Academy of Sciences, China
Jungang Xu, University of Chinese Academy of Sciences, China
Yue Liu, Shanghai University, China
Zhen Jia, Amazon, USA