<div dir="ltr">Probably matrix assembly on GPU is more important. Do you have an example for me to play to see what GPU interface we should have?<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 5:44 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I was looking into assembling matrices with threads. I have a coloring to avoid conflicts.<div><br></div><div>Turning off all the logging seems way overkill and for methods that can get called in a thread then we could use PETSC_HAVE_THREADSAFTEY (thingy) to protect logging functions. So one can still get timings for the whole assembly process, just not for MatSetValues. Few people are going to do this. I don't think it will be a time sink, and if it is we just revert back to saying 'turn logging off'. I don't see a good argument for insisting on turning off logging, it is pretty important, if we just say that we are going to protect methods as needed.</div><div><br></div><div>It is not a big deal, I am just exploring this idea. It is such a basic concept in shared memory sparse linear algebra that it seems like a good thing to be able to support and have in an example to say we can assemble matrices in threads (not that it is a great idea). We have all the tools (eg, coloring methods) that it is just a matter of protecting code a few methods. I use DMPlex MatClosure instead of MatSetValues and this is where I die now with non-thread safe code. We have an idea, from Jed, on how to fix it.</div><div><br></div><div>Anyway, thanks for your help, but I think we should hold off on doing anything until we have some consensus that this would be a good idea to put some effort into getting a thread safe PETSc that can support OMP matrix assembly with a nice compact example.</div><div><br></div><div>Thanks again,</div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 5:44 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mark,<div> I saw you had "--with-threadsaftey --with-log=0". Do you really want to call petsc from multiple threads (in contrast to letting petsc call other libraries, e.g., BLAS, doing multithreading)? If not, you can drop --with-threadsaftey.</div><div> I have <a href="https://gitlab.com/petsc/petsc/-/merge_requests/2714" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/2714</a> that should fix your original compilation errors.</div><div><br></div><div><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 2:07 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><a href="https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html" target="_blank">https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html</a><br><div><br></div><div>and I see this on my Mac:</div><div><br></div><div>14:23 1 mark/feature-xgc-interface-rebase *= ~/Codes/petsc$ ../arch-macosx-gnu-O-omp.py <br>===============================================================================<br> Configuring PETSc to compile on your system <br>===============================================================================<br>=============================================================================== Warning: PETSC_ARCH from environment does not match command-line or name of script. Warning: Using from command-line or name of script: arch-macosx-gnu-O-omp, ignoring environment: arch-macosx-gnu-g =============================================================================== TESTING: configureLibraryOptions from PETSc.options.libraryOptions(config/PETSc/options/libraryOptions.py:37) *******************************************************************************<br> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):<br>-------------------------------------------------------------------------------<br>Must use --with-log=0 with --with-threadsafety<br>*******************************************************************************<br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 2:54 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 12:06 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">BTW, I can build on SUMMIT with logging and OMP, apparently. I also seem to be able to build with debugging. Both of which are not allowed according the the docs. I am puzzled.</div></blockquote><div> What are "the docs"?</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 12:05 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I think the problem is that you have to turn off logging with openmp and the (newish) GPU timers did not protect their timers.<div><br></div><div>I don't see a good reason to require logging be turned off with OMP. We could use PETSC_HAVE_THREADSAFETY to protect logs that we care about (eg, in MatSetValues) and as users discover more things that they want to call in an OMP thread block, then tell them to turn logging off and we will fix it when we can.</div><div><br></div><div>Any thoughts on the idea of letting users keep logging with openmp?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 11:40 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes. Looks we need to include petsclog.h. Don't know why OMP triggered the error.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 13, 2020 at 9:59 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Should I do an MR to fix this?</div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>