<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 21, 2021, at 5:37 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">This did not work. I verified that MPI_Init_thread is being called correctly and that MPI returns that it supports this highest level of thread safety.<div class=""><br class=""></div><div class="">I am going to ask ORNL. <div class=""><br class=""></div><div class="">And if I use:</div><div class=""><br class=""><div class="">-fieldsplit_i1_ksp_norm_type none<br class="">-fieldsplit_i1_ksp_max_it 300</div><div class=""><br class=""></div><div class="">for all 9 "i" variables, I can run normal iterations on the 10th variable, in a 10 species problem, and it works perfectly with 10 threads.</div><div class=""><br class=""></div><div class="">So it is definitely that VecNorm is not thread safe.<br class=""><div class=""><br class=""></div><div class="">And, I want to call SuperLU_dist, which uses threads, but I don't want SuperLU to start using threads. Is there a way to tell superLU that there are no threads but have PETSc use them?</div></div></div></div></div></div></blockquote><div><br class=""></div> My interpretation and Satish's for many years is that SuperLU_DIST has to be built with and use OpenMP in order to work with CUDA. </div><div><br class=""></div><div><div> def formCMakeConfigureArgs(self):</div><div> args = config.package.CMakePackage.formCMakeConfigureArgs(self)</div><div> if self.openmp.found:</div><div> self.usesopenmp = 'yes'</div><div> else:</div><div> args.append('-DCMAKE_DISABLE_FIND_PACKAGE_OpenMP=TRUE')</div><div> if self.cuda.found:</div><div> if not self.openmp.found:</div><div> raise RuntimeError('SuperLU_DIST GPU code currently requires OpenMP. Use --with-openmp=1')</div><div><br class=""></div><div>But this could be ok. You use OpenMP and then it uses OpenMP internally, each doing their own business (what could go wrong :-)).</div><div><br class=""></div><div>Have you tried it?</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark</div></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 21, 2021 at 5:19 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">OK, the problem is probably:<div class=""><br class=""><div class="">PetscMPIInt PETSC_MPI_THREAD_REQUIRED = MPI_THREAD_FUNNELED;<br class=""></div></div><div class=""><br class=""></div><div class="">There is an example that sets:</div><div class=""><br class=""></div><div class="">PETSC_MPI_THREAD_REQUIRED = MPI_THREAD_MULTIPLE;<br class=""></div><div class=""><br class=""></div><div class="">This is what I need.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 21, 2021 at 2:26 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 21, 2021 at 2:11 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jan 21, 2021 at 2:02 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jan 21, 2021 at 1:44 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jan 21, 2021 at 11:16 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Yes, the problem is that each KSP solver is running in an OMP thread (So at this point it only works for SELF and its Landau so it is all I need). It looks like MPI reductions called with a comm_self are not thread safe (eg, the could say, this is one proc, thus, just copy send --> recv, but they don't)</div></blockquote><div class=""><br class=""></div><div class="">Instead of using SELF, how about Comm_dup() for each thread?</div></div></div></blockquote><div class=""><br class=""></div><div class="">OK, raw MPI_Comm_dup. I tried PetscCommDup. Let me this.</div><div class="">Thanks, </div></div></div></blockquote><div class=""><br class=""></div><div class="">You would have to dup them all outside the OMP section, since it is not threadsafe. Then each thread uses one I think.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Yea sure. I do it in SetUp.</div><div class=""><br class=""></div><div class="">Well that worked to get <b class="">different Comms</b>, finally, I still get the same problem. The number of iterations differ wildly. This two species and two threads (13 SNES its that is not deterministic). Way below is one thread (8 its) and fairly uniform iteration counts.</div><div class=""><br class=""></div><div class="">Maybe this MPI is just not thread safe at all. Let me look into it.</div><div class="">Thanks anyway,</div><div class=""><br class=""></div><div class=""> 0 SNES Function norm 4.974994975313e-03<br class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x80017c60. Comms pc=0x67ad27c0 ksp=<b class="">0x7ffe1600</b> newcomm=0x8014b6e0<br class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x7ffdabc0. Comms pc=0x67ad27c0 ksp=<b class="">0x7fff70d0</b> newcomm=0x7ffe9980<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 282<br class=""> 1 SNES Function norm 1.836376279964e-05<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 19<br class=""> 2 SNES Function norm 3.059930074740e-07<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 15<br class=""> 3 SNES Function norm 4.744275398121e-08<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 4<br class=""> 4 SNES Function norm 4.014828563316e-08<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 456<br class=""> 5 SNES Function norm 5.670836337808e-09<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 2<br class=""> 6 SNES Function norm 2.410421401323e-09<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 18<br class=""> 7 SNES Function norm 6.533948191791e-10<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 458<br class=""> 8 SNES Function norm 1.008133815842e-10<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 9<br class=""> 9 SNES Function norm 1.690450876038e-11<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 4<br class=""> 10 SNES Function norm 1.336383986009e-11<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 463<br class=""> 11 SNES Function norm 1.873022410774e-12<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 113<br class=""> 12 SNES Function norm 1.801834606518e-13<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_ATOL iterations 1<br class=""> 13 SNES Function norm 1.004397317339e-13<br class=""> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 13<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> 0 SNES Function norm 4.974994975313e-03<br class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x6e265010. Comms pc=0x56450340 ksp=0x6e2168d0 newcomm=0x6e265090<br class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x6e25bc40. Comms pc=0x56450340 ksp=0x6e22c1d0 newcomm=0x6e21e8f0<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 282<br class=""> 1 SNES Function norm 1.836376279963e-05<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 380<br class=""> 2 SNES Function norm 3.018499983019e-07<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 387<br class=""> 3 SNES Function norm 1.826353175637e-08<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 391<br class=""> 4 SNES Function norm 1.378600599548e-09<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 392<br class=""> 5 SNES Function norm 1.077289085611e-10<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 394<br class=""> 6 SNES Function norm 8.571891727748e-12<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 395<br class=""> 7 SNES Function norm 6.897647643450e-13<br class=""> Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 395<br class=""> 8 SNES Function norm 5.606434614114e-14<br class=""> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 8<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 21, 2021 at 10:46 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jan 21, 2021 at 10:34 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">It looks like PETSc is just too clever for me. I am trying to get a different MPI_Comm into each block, but PETSc is thwarting me:</div></blockquote><div class=""><br class=""></div><div class="">It looks like you are using SELF. Is that what you want? Do you want a bunch of comms with the same group, but independent somehow? I am confused.</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""> if (jac->use_openmp) {<br class=""> ierr = KSPCreate(MPI_COMM_SELF,&ilink->ksp);CHKERRQ(ierr);<br class="">PetscPrintf(PETSC_COMM_SELF,"In PCFieldSplitSetFields_FieldSplit with -------------- link: %p. Comms %p %p\n",ilink,PetscObjectComm((PetscObject)pc),PetscObjectComm((PetscObject)ilink->ksp));<br class=""> } else {<br class=""> ierr = KSPCreate(PetscObjectComm((PetscObject)pc),&ilink->ksp);CHKERRQ(ierr);<br class=""> }<br class=""></div><div class=""><br class=""></div><div class="">produces:</div><div class=""><br class=""></div><div class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x7e9cb4f0. Comms 0x660c6ad0 0x660c6ad0<br class="">In PCFieldSplitSetFields_FieldSplit with -------------- link: 0x7e88f7d0. Comms 0x660c6ad0 0x660c6ad0<br class=""></div><div class=""><br class=""></div><div class="">How can I work around this?</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 21, 2021 at 7:41 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 20, 2021 at 6:21 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 20, 2021, at 3:09 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">So I put in a temporary hack to get the first Fieldsplit apply to NOT use OMP and it sort of works. <div class=""><br class=""></div><div class="">Preonly/lu is fine. GMRES calls vector creates/dups in every solve so that is a big problem. </div></div></div></blockquote><div class=""><br class=""></div> It should definitely not be creating vectors "in every" solve. But it does do lazy allocation of needed restarted vectors which may make it look like it is creating "every" vectors in every solve. You can use -ksp_gmres_preallocate to force it to create all the restart vectors up front at KSPSetUp(). </div></div></blockquote><div class=""><br class=""></div><div class="">Well, I run the first solve w/o OMP and I see Vec dups in cuSparse Vecs in the 2nd solve. </div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class=""> Why is creating vectors "at every solve" a problem? It is not thread safe I guess?</div></div></blockquote><div class=""><br class=""></div><div class="">It dies when it looks at the options database, in a Free in the get-options method to be exact (see stacks). </div><div class=""><br class=""></div><div class="">======= Backtrace: =========<br class="">/lib64/libc.so.6(cfree+0x4a0)[0x200021839be0]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PetscFreeAlign+0x4c)[0x2000002a368c]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PetscOptionsEnd_Private+0xf4)[0x2000002e53f0]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x7c6c28)[0x2000008b6c28]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecCreate_SeqCUDA+0x11c)[0x20000052c510]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecSetType+0x670)[0x200000549664]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecCreateSeqCUDA+0x150)[0x20000052c0b0]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x43c198)[0x20000052c198]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicate+0x44)[0x200000542168]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicateVecs_Default+0x148)[0x200000543820]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(VecDuplicateVecs+0x54)[0x2000005425f4]<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPCreateVecs+0x4b4)[0x2000016f0aec]<br class=""></div><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="">Richardson works except the convergence test gets confused, presumably because MPI reductions with PETSC_COMM_SELF is not threadsafe.</div></div></div></blockquote><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">One fix for the norms might be to create each subdomain solver with a different communicator.</div></div></div></blockquote><div class=""><br class=""></div> Yes you could do that. It might actually be the correct thing to do also, if you have multiple threads call MPI reductions on the same communicator that would be a problem. Each KSP should get a new MPI_Comm. </div></div></blockquote><div class=""><br class=""></div><div class="">OK. I will only do this.</div><div class=""><br class=""></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></body></html>