<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Hong<div class=""><br class=""></div><div class=""> Thanks.</div><div class=""><br class=""></div><div class=""> What else is needed to get all of GAMG setup onto the GPU and running as well as it can? Any more GPU kernels? Or is just tweaking the infrastucture, the kernels are already done? Will we see pure GPU GAMG soon?</div><div class=""><br class=""></div><div class=""> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. </div><div class=""><br class=""></div><div class=""> Others?</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""> And no, this is not the time for arguing GPUs suck and some things are best done on CPUs or that DOE should not have bought GPU machines. It is about getting the best performance we can from a set of systems we will have available.</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 12, 2021, at 5:43 PM, Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 12, 2021, at 5:25 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
Jed,
<div class=""><br class="">
</div>
<div class=""> Thanks for the insight.</div>
<div class=""><br class="">
</div>
<div class=""> Maybe Hong and his Ellpack format? Or his independent set algorithm?</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">These two features are currently functional on NVIDIA GPUs. Neither needs extensive development or refactorization, thus not suitable for this Argonne GPU Hackathon. But we do need reviewers to look at the two MRs and move forward. </div>
<div class=""><br class="">
</div>
<div class="">It is also worth mentioning that the CUDA-based maximal independent set algorithm was developed during last GPU Hackathon at ORNL. Richard, Junchao and I attended that one. It was a great experience. The commitment is extremely intensive for both the mentors
and the attendees. </div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Hong </div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class=""> Maybe Stefano and his COO matrix assembly on GPUs?</div>
<div class=""><br class="">
</div>
<div class=""> Others?</div>
<div class=""><br class="">
</div>
<div class="">Barry</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 12, 2021, at 4:37 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I
helped with one of these a couple years ago. It's important to go in with a well-defined problem and mini-app. If "PETSc" is the topic, then you should start with a representative benchmark problem that runs in no more than a couple minutes. It could be two
problems, one that we think is good and one that we think is weak. The format is good for incrementally profiling and refactoring existing code. Probably less so for doing careful implementations of genuinely novel algorithms. (And if you plan to do that,
make sure there's a way that the one person focused on that lift doesn't block the rest of the team.)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Barry
Smith <</span><a href="mailto:bsmith@petsc.dev" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">bsmith@petsc.dev</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">>
writes:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Folks,<br class="">
<br class="">
Get was able to get us into the ANL GPU hackathon that will use the "theta" GPU system at ANL (that has NVIDIA GPUs). ALCF accounts are needed for the hackathon (but can be created). Registration information is below. If we go with specific goals it could
be a useful experience.<br class="">
<br class="">
Barry<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Begin forwarded message:<br class="">
<br class="">
From: Andi Moore <<a href="mailto:amoore@nvidia.com" class="">amoore@nvidia.com</a>><br class="">
Subject: Argonne GPU Virtual Hackathon - Accepted<br class="">
Date: March 11, 2021 at 5:09:02 PM CST<br class="">
To: "<a href="mailto:d.getnet@gmail.com" class="">d.getnet@gmail.com</a>" <<a href="mailto:d.getnet@gmail.com" class="">d.getnet@gmail.com</a>>, "<a href="mailto:gbetrie@anl.gov" class="">gbetrie@anl.gov</a>" <<a href="mailto:gbetrie@anl.gov" class="">gbetrie@anl.gov</a>>,
"<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>" <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class="">
Cc: "Ghadar, Yasaman" <<a href="mailto:ghadar@anl.gov" class="">ghadar@anl.gov</a>>, Julia Levites <<a href="mailto:jlevites@nvidia.com" class="">jlevites@nvidia.com</a>><br class="">
Message-Id: <<a href="mailto:MW3PR12MB4588BF988609B8F7A49F6A21B3909@MW3PR12MB4588.namprd12.prod.outlook.com" class="">MW3PR12MB4588BF988609B8F7A49F6A21B3909@MW3PR12MB4588.namprd12.prod.outlook.com</a>><br class="">
<br class="">
Dear Team Barry Smith,<br class="">
<br class="">
Congratulations! Your team has been selected to participate in the Argonne GPU Virtual Hackathon.<span class="Apple-converted-space"> </span><br class="">
<br class="">
The next steps:<br class="">
<br class="">
Registration<br class="">
Each team member is required to register for the event. Please click here <<a href="https://www.gpuhackathons.org/form/gpu-event-registration" class="">https://www.gpuhackathons.org/form/gpu-event-registration</a>> to register and select the Hackathon from
the drop-down menu. Please register as soon as possible.<br class="">
<br class="">
Mentors<br class="">
Mentors are being selected for each team based on your research fields and coding needs. An introductory call for teams and mentors has been scheduled for Wednesday, April 7th, details below.<span class="Apple-converted-space"> </span><br class="">
<br class="">
System Access<br class="">
Request an ALCF Computer User Account <<a href="https://accounts.alcf.anl.gov/" class="">https://accounts.alcf.anl.gov/</a>> if you do not currently have one. If you have an ALCF account, but it is currently inactive, reactivate <<a href="https://accounts.alcf.anl.gov/#!/accountReactivate" class="">https://accounts.alcf.anl.gov/#!/accountReactivate</a>>
your account.<span class="Apple-converted-space"> </span><br class="">
Specify the following in the account request/reactivation form:<br class="">
Project Name: gpu_hack<br class="">
Principal Investigator (PI): Yasaman Ghadar<br class="">
<br class="">
Contact<span class="Apple-converted-space"> </span><a href="mailto:accounts@alcf.anl.gov" class="">accounts@alcf.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:accounts@alcf.anl.gov" class="">mailto:accounts@alcf.anl.gov</a>> if you
have questions about your ALCF account.<br class="">
<br class="">
Additional information<br class="">
Review the GPU Hackathon Virtual Attendee Guide <<a href="https://docs.google.com/document/d/1M7vacg0GLg38gsJYHOg2tM_M0QArOSCrPiLaQER9hzg/edit?usp=sharing" class="">https://docs.google.com/document/d/1M7vacg0GLg38gsJYHOg2tM_M0QArOSCrPiLaQER9hzg/edit?usp=sharing</a>><br class="">
Review the technical resources website <<a href="https://www.gpuhackathons.org/technical-resources" class="">https://www.gpuhackathons.org/technical-resources</a>> which will help you prepare for the hackathon. The page includes information about profiling
tools, compilers, libraries, programming models and more.<span class="Apple-converted-space"> </span><br class="">
<br class="">
Slack<br class="">
All communication for the event will occur through the Slack workspace<br class="">
Please join <<a href="https://join.slack.com/t/argonnehack2021/shared_invite/zt-nfio2vwz-f0UeYdXVmoRLKUIBgLy_FA" class="">https://join.slack.com/t/argonnehack2021/shared_invite/zt-nfio2vwz-f0UeYdXVmoRLKUIBgLy_FA</a>> the Slack workspace<span class="Apple-converted-space"> </span><br class="">
<br class="">
Agenda - All calendar invites are attached<span class="Apple-converted-space"> </span><br class="">
Argonne GPU Hackathon Team and Mentor Meeting: 2:00PM-3:00PM CST – April 7, 2021 Zoom Link <<a href="https://zoom.us/j/93527581258?pwd=S01GZVAxS3hLVCsvc0wyR2s2MHJEZz09" class="">https://zoom.us/j/93527581258?pwd=S01GZVAxS3hLVCsvc0wyR2s2MHJEZz09</a>><br class="">
<br class="">
Argonne GPU Hackathon Day 1: 9:00AM-5:00PM CST – April 20, 2021 Zoom Link <<a href="https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09" class="">https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09</a>><br class="">
<br class="">
Argonne GPU Hackathon Day 2: 9:00AM-5:00PM CST April 27, 2021 Zoom Link <<a href="https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09" class="">https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09</a>><br class="">
<br class="">
Argonne GPU Hackathon Day 3: 9:00AM-5:00PM CST April 28, 2021 Zoom Link <<a href="https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09" class="">https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09</a>><br class="">
<br class="">
Argonne GPU Hackathon Day 4: 9:00AM-5:30PM CST April 29, 2021 Zoom Link <<a href="https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09" class="">https://zoom.us/j/94645872736?pwd=UmtNakprSjB6WmFVUVdoZUMvdDFIUT09</a>><br class="">
<br class="">
<br class="">
Please feel free to ask any questions you may have. Thank you all for your interest and outstanding applications!<span class="Apple-converted-space"> </span><br class="">
<br class="">
Best regards,<br class="">
The GPU Hackathon Team<span class="Apple-converted-space"> </span></blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div></blockquote></div><br class=""></div></body></html>