<div dir="ltr">I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.<div><br></div><div><br></div><div>In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0:<br>/ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure'<br> typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure;<br> ^~~~~~~~~~~~~~~~~~~~~~~~~~<br>/ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here<br> typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure;<br> ^~~~~~~~~~~~~~~~~~~~~~~~~~<br> CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present)<div>Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?</div><div><div><br><blockquote type="cite"><div>On May 28, 2021, at 8:44 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;"><div><br></div><div> Stefano, who has a far better memory than me, wrote</div><div><br></div>> Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?<div><br></div><div> MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages </div><div><br></div><div> but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.</div><div><br></div><div> Barry</div><div><br><div><br><div><br><blockquote type="cite"><div>On May 28, 2021, at 12:41 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">OK, I will try to rebase and test Barry's branch.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice<div>You can use both I guess<br><div><br><blockquote type="cite"><div>On May 28, 2021, at 8:25 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr"><div>Is this the correct branch? It conflicted with ex5cu so I assume it is.</div><div><br></div><div><br></div><a href="https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device" style="box-sizing:border-box;color:rgb(48,48,48);text-decoration-line:none;margin-left:0.5rem;display:inline-block;overflow:hidden;text-overflow:ellipsis;vertical-align:top;white-space:nowrap;max-width:100%;font-weight:600;font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13.3px" target="_blank">stefanozampini/simplify-setvalues-device</a><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:24 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I am fixing rebasing this branch over main.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?<br><div><br><blockquote type="cite"><div>On May 28, 2021, at 8:12 PM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:</div><br><div><div>That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?<br><div><br><blockquote type="cite"><div>On May 28, 2021, at 7:45 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div><div><br></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">~/petsc/src/mat/tutorials</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)"><b> (barry/2021-05-28/robustify-cuda-gencodearch-check=)</b></span><span style="font-variant-ligatures:no-common-ligatures"> arch-robustify-cuda-gencodearch-check</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">$ ./ex5cu</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">terminate called after throwing an instance of 'thrust::system::system_error'</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Aborted (core dumped)</span></div><div><br></div><div><span style="font-variant-ligatures:no-common-ligatures"> requires: cuda !define(PETSC_USE_CTABLE)</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"> CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"> Barry</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><br><blockquote type="cite"><div>On May 28, 2021, at 11:13 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>If you are referring to your device set values, I guess it is not currently tested</div></blockquote><div><br></div><div>No. There is a test for that (ex5cu).</div><div>I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases.</div><div>I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>See the discussions here <a href="https://gitlab.com/petsc/petsc/-/merge_requests/3411" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/3411</a><br><div>I started cleaning up the code to prepare for testing but we never finished it <a href="https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/" target="_blank">https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/</a></div><div><br><div><br><blockquote type="cite"><div>On May 28, 2021, at 6:53 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">Is there a test with MatSetValues and CUDA? </div>
</div></blockquote></div><br></div></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br></div></div></blockquote></div><br></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div></blockquote></div>