<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 2, 2021, at 2:06 AM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Barry<div class=""><br class=""></div><div class="">If I remember correctly, the ctable business was a limitation of Mark's initial implementation, which I removed here <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3411/diffs?commit_id=36173707431f5583889720868beae9046e6e1fd2" class="">https://gitlab.com/petsc/petsc/-/merge_requests/3411/diffs?commit_id=36173707431f5583889720868beae9046e6e1fd2</a>. Here colmap was "int" already. We made a mistake letting this branch fall so behind main. I will add my observations in the MR</div></div></div></blockquote><div><br class=""></div> Thanks, likely it was the merge brought in the incorrect colmap. </div><div><br class=""></div><div> Yes, it was my fault the branch go neglected, I become too emotional over the configure issue with CUDA, stating it needed a completely rewrite and then forgot about the branch. I have "fixed" the CUDA configure issue with an "ok" but hacky manner, after MPI is configured and changes the compilers, thus removing all the compiler flags, I force a rerun of parts of cuda.py to add back their contributions to the CUDA compiler flags. There is a circular dependency that this change resolves. </div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 2 giu 2021 alle ore 09:20 Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> Switching the thread to petsc-dev since users don't need this.<div class=""><br class=""></div><div class=""> Had a nasty failure on one the CI, turned out the GPU colmap was declared as PetscInt* but it was copied down from the CPU into the GPU memory as int. The power of void* interface. Thus <a href="http://ex5cu.cu/" target="_blank" class="">ex5cu.cu</a> which was never CI tested was crashing very strangely. By declaring it properly as int* I got the example to run with 64 bit indices. Submitted a new pipeline to see what else pops up. </div><div class=""><br class=""></div><div class=""> I am a bit worried about the csr2csc data structures and code, one line declaring csr2csc_i inside the struct disappeared in the rebase on main, I stuck it in but don't understand that code at all. I'll report further when the CI is done or crashes. I will also need to make sure it is compiled pre and post PETSC_PKG_CUDA_VERSION_GE(11,0,0)</div><div class=""><br class=""></div><div class=""> I will also investigate this CTABLE business, the GPU doesn't need to know or care if the CPU is using a CTABLE or not; it just needs the colmap array format that it gets from spreading out the garray anyway to be copied to the GPU so PETSc can likely be built with CTABLES as normal; this will improve the testing a lot. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Barry</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 30, 2021, at 11:54 AM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> I believe I have finally successfully rebased the branch <span style="font-family:Menlo;font-size:14px" class="">barry/2020-11-11/cleanup-matsetvaluesdevice </span>against main and cleaned up all the issues. Please read the commit message.<div class=""><br class=""></div><div class=""> I have submitted a CI pipeline with ctables turned off temporarily for testing of the MatSetValuesDevice(). If it works hopefully Mark can maybe run a few additional tests of his Landau code that are not in the usual testing to verify and we can finally get the branch into main.</div><div class=""><br class=""></div><div class=""> Mark,</div><div class=""><br class=""></div><div class=""> Since this change is involved, it is likely your Landau mass matrix branch may not rebase cleanly. Let me know if you would like me to do the rebase and testing of your Landau mass matrix branch. I can get it ready to work with the results of <span style="font-family:Menlo;font-size:14px" class="">barry/2020-11-11/cleanup-matsetvaluesdevice </span>and then hand it back to you for further development.</div><div class=""><font face="Menlo" class=""><span style="font-size:14px" class=""><br class=""></span></font></div><div class=""><font face="Menlo" class=""><span style="font-size:14px" class=""> Barry</span></font></div><div class=""><font face="Menlo" class=""><span style="font-size:14px" class=""><br class=""></span></font><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 29, 2021, at 11:32 AM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> I am working away on this branch, making some progress, also cleaning things up with some small simplifications. Hope I can succeed, a bunch of stuff got moved around and some structs had changes, the merge could not handle some of these so I have to do a good amount of code wrangling to fix it.<div class=""><br class=""></div><div class=""> I'll let you know as I progress.<br class=""><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 10:53 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> I have rebased and tried to fix everything. I am now fixing the issues of --download-openmpi and cuda, once that is done I will test, rebase with main again if needed and restart the MR and get it into main.<div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class="">I was stupid to let the MR lay fallow, I should have figured out a solution to the openmpi and cuda issue instead of punting and waiting for a dream fix.</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 2:39 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Thanks,<div class=""><br class=""><div class="">I did not intend to make any (real) changes. </div><div class="">The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of <a href="http://ex5cu.cu/" target="_blank" class="">ex5cu.cu</a></div><div class=""><br class=""></div><div class="">* I ended up with two declarations of PetscSplitCSRDataStructure</div><div class="">* I added some includes to fix errors like this:</div><div class="">/ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed<br class=""></div><div class="">* I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: </div><div class="">/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu/" target="_blank" class="">aijcusparse.cu</a>(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i"<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">I can take a quick look at it tomorrow, what are the main changes you made since then?<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 9:51 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">I am getting messed up in trying to resolve conflicts in rebasing over main.<div class="">Is there a better way of doing this?</div><div class="">Can I just tell git to use Barry's version and then test it?</div><div class="">Or should I just try it again?</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 2:15 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.<div class=""><br class=""></div><div class=""><br class=""></div><div class="">In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0:<br class="">/ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure'<br class=""> typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure;<br class=""> ^~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">/ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here<br class=""> typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure;<br class=""> ^~~~~~~~~~~~~~~~~~~~~~~~~~<br class=""> CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present)<div class="">Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 8:44 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div class=""><div class=""><br class=""></div><div class=""> Stefano, who has a far better memory than me, wrote</div><div class=""><br class=""></div>> Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?<div class=""><br class=""></div><div class=""> MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages </div><div class=""><br class=""></div><div class=""> but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 12:41 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">OK, I will try to rebase and test Barry's branch.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice<div class="">You can use both I guess<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 8:25 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">Is this the correct branch? It conflicted with ex5cu so I assume it is.</div><div class=""><br class=""></div><div class=""><br class=""></div><a href="https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device" style="box-sizing:border-box;color:rgb(48,48,48);text-decoration-line:none;margin-left:0.5rem;display:inline-block;overflow:hidden;text-overflow:ellipsis;vertical-align:top;white-space:nowrap;max-width:100%;font-weight:600;font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13.3px" target="_blank" class="">stefanozampini/simplify-setvalues-device</a><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:24 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I am fixing rebasing this branch over main.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 8:12 PM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:</div><br class=""><div class=""><div class="">That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 7:45 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:</div><br class=""><div class=""><div class=""><div class=""><br class=""></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">~/petsc/src/mat/tutorials</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)" class=""><b class=""> (barry/2021-05-28/robustify-cuda-gencodearch-check=)</b></span><span style="font-variant-ligatures:no-common-ligatures" class=""> arch-robustify-cuda-gencodearch-check</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ ./ex5cu</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">terminate called after throwing an instance of 'thrust::system::system_error'</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Aborted (core dumped)</span></div><div class=""><br class=""></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> requires: cuda !define(PETSC_USE_CTABLE)</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Barry</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 11:13 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">If you are referring to your device set values, I guess it is not currently tested</div></blockquote><div class=""><br class=""></div><div class="">No. There is a test for that (ex5cu).</div><div class="">I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases.</div><div class="">I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="">See the discussions here <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3411" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/3411</a><br class=""><div class="">I started cleaning up the code to prepare for testing but we never finished it <a href="https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/" target="_blank" class="">https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/</a></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 28, 2021, at 6:53 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Is there a test with MatSetValues and CUDA? </div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></div></div></blockquote></div><br class=""></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature">Stefano</div>
</div></blockquote></div><br class=""></body></html>