<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> I submit it is actually a good amount of additional work and requires real creativity and very good judgment; it is not a good intro or undergrad project; especially for someone without a huge amount of hands-on experience already. Look who had to do the new SpecHPC multigrid benchmark. The last time I checked Sam was not an undergrad. <span style="caret-color: rgb(77, 81, 86); color: rgb(77, 81, 86); font-family: arial, sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Senior Scientist, Lawrence </span><span style="font-weight: bold; color: rgb(95, 99, 104); font-family: arial, sans-serif; font-size: 14px;" class="">Berkeley</span><span style="caret-color: rgb(77, 81, 86); color: rgb(77, 81, 86); font-family: arial, sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);" class=""> National Laboratory - Cited by 11194</span> I definitely do not plan to involve myself in any brand new serious benchmarking studies in my current lifetime, doing one correctly is a massive undertaking IMHO.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 22, 2022, at 6:43 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">This isn't so much more or less work, but work in more useful places. Maybe this is a good undergrad or intro project to make a clean workflow for these experiments.<br class=""><br class="">Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> writes:<br class=""><br class=""><blockquote type="cite" class=""> Performance studies are enormously difficult to do well; which is why there are so few good ones out there. And unless you fall into the LINPACK benchmark or hit upon Streams the rewards of doing an excellent job are pretty thin. Even Streams was not properly maintained for many years, you could not just get it and use it out of the box for a variety of purposes (which is why PETSc has its hacked-up ones). I submit a properly performance study is a full-time job and everyone always has those.<br class=""><br class=""><blockquote type="cite" class="">On Jan 22, 2022, at 2:11 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:<br class=""><br class="">Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> writes:<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">On Jan 22, 2022, at 12:15 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:<br class="">Barry, when you did the tech reports, did you make an example to reproduce on other architectures? Like, run this one example (it'll run all the benchmarks across different sizes) and then run this script on the output to make all the figures?<br class=""></blockquote><br class=""> It is documented in <a href="https://www.overleaf.com/project/5ff8f7aca589b2f7eb81c579" class="">https://www.overleaf.com/project/5ff8f7aca589b2f7eb81c579</a> You may need to dig through the submit scripts etc to find out exactly.<br class=""></blockquote><br class="">This runs a ton of small jobs and each job doesn't really preload, but instead of loops in job submission scripts, the loops could be inside the C code and it could directly output tabular data. This would run faster and be easier to submit and analyze.<br class=""><br class=""><a href="https://gitlab.com/hannah_mairs/summit-performance/-/blob/master/summit-submissions/submit_gpu1.lsf" class="">https://gitlab.com/hannah_mairs/summit-performance/-/blob/master/summit-submissions/submit_gpu1.lsf</a><br class=""><br class="">It would hopefully also avoid writing the size range manually over here in the analysis script where it has to match exactly the job submission.<br class=""><br class="">https://gitlab.com/hannah_mairs/summit-performance/-/blob/master/python/graphs.py#L8-9<br class=""><br class=""><br class="">We'd make our lives a lot easier understanding new machines if we put into the design of performance studies just a fraction of the kind of thought we put into public library interfaces.<br class=""></blockquote></blockquote></div></div></blockquote></div><br class=""></body></html>