<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div> Good. I have cleaned up the code and rebased the branch so if you use it in the future you need to do</div><div><br class=""></div><div> git fetch</div><div> git checkout master </div><div> git branch -D <b style="color: rgb(200, 20, 201); font-family: Menlo; font-size: 15px;" class="">barry/2020-07-28/superlu_dist-single</b></div><div><font color="#c814c9" face="Menlo" class=""><span style="caret-color: rgb(200, 20, 201); font-size: 15px;" class=""><b class=""> </b></span></font>git checkout <b style="color: rgb(200, 20, 201); font-family: Menlo; font-size: 15px;" class="">barry/2020-07-28/superlu_dist-single</b></div><div><b style="color: rgb(200, 20, 201); font-family: Menlo; font-size: 15px;" class=""> </b></div><div> Do not just do a git pull in <b style="color: rgb(200, 20, 201); font-family: Menlo; font-size: 15px;" class="">barry/2020-07-28/superlu_dist-single </b>that will make a mess</div><div><font color="#c814c9" face="Menlo" class=""><span style="caret-color: rgb(200, 20, 201); font-size: 15px;" class=""><b class=""><br class=""></b></span></font></div><div><font color="#c814c9" face="Menlo" class=""><span style="caret-color: rgb(200, 20, 201); font-size: 15px;" class=""><b class=""><br class=""></b></span></font><blockquote type="cite" class=""><div class="">On Aug 4, 2020, at 8:09 AM, <a href="mailto:flw@rzg.mpg.de" class="">flw@rzg.mpg.de</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">PS:<br class="">I saw you committed a new version for the SUPERLU_DIST installer.<br class="">There is a missing ")" at the end of line 42.<br class="">Also I think one also has to add<br class="">self.addToArgs(args,'-DCMAKE_CXX_FLAGS:STRING','-I'+self.blasLapack.include)<br class=""><br class="">as well (for me only adding the C flag raised the same error as before; with this, it works for me even if CPATH isn't set)<br class="">Thank you again very much :)<br class=""><br class="">Best regards,<br class="">Felix<br class=""><br class="">Zitat von Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>>:<br class=""><br class=""><blockquote type="cite" class="">On some systems, like my Mac, the environmental variable CPATH is set to the appropriate directory for mkl.h when MKL is initialized with<br class=""><br class="">source /opt/intel/compilers_and_libraries/mac/bin/compilervars.sh intel64<br class=""><br class="">hence the compiler can magically find mkl.h but this doesn't seem to be universal, I think that is why Felix's system cannot find the include, I found a Linux machine at ANL that produces the same problem Felix has. Anyways I'm fixing the PETSc install will just work even if CPATH is not set.<br class=""><br class=""> Barry<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On Aug 3, 2020, at 10:21 PM, Xiaoye S. Li <<a href="mailto:xsli@lbl.gov" class="">xsli@lbl.gov</a>> wrote:<br class=""><br class="">Regarding MKL, superlu only uses some BLAS routines in MKL. If you have alternative BLAS, you don't need to use MKL.<br class=""><br class="">But, in my experience, on an Intel system, the compiler can correctly include the path with mkl.h, that is, I don't need to do anything special.<br class="">I think mpiicc should work.<br class=""><br class="">Sherry<br class=""><br class="">On Mon, Aug 3, 2020 at 8:46 AM <<a href="mailto:flw@rzg.mpg.de" class="">flw@rzg.mpg.de</a> <<a href="mailto:flw@rzg.mpg.de" class="">mailto:flw@rzg.mpg.de</a>>> wrote:<br class="">Hi Barry,<br class="">Thanks for the branch (and thanks to Sherry as well). I tried to use<br class="">the configure example " arch-ci-linux-intel-mkl-single.py" (adding the<br class="">--with-batch flag, since I am running on a cluster), but I get the<br class="">following error message:<br class="">TESTING: check from<br class="">config.libraries(config/BuildSystem/config/libraries.py:157)<br class=""><br class=""><br class=""><br class="">*******************************************************************************<br class=""> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log<br class="">for details):<br class="">-------------------------------------------------------------------------------<br class="">Downloaded ptscotch could not be used. Please check install in<br class="">/u/flw/petsc-hash-pkgs/073aec050<br class=""><br class="">(see configure1.log)<br class=""><br class="">Next I tried a minimalistic version using Intel MPI compilers:<br class="">./configure --download-superlu_dist --download-metis<br class="">--download-parmetis --download-ptscotch --with-precision=single<br class="">--with-batch --with-fc=mpiifort --with-cc=mpiicc --with-cxx=mpiicpc<br class=""><br class="">There I got the following error:<br class="">=============================================================================== Compiling and installing SUPERLU_DIST; this may take several minutes ===============================================================================<br class="">*******************************************************************************<br class=""> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log<br class="">for details):<br class="">-------------------------------------------------------------------------------<br class="">Error running make on SUPERLU_DIST<br class=""><br class="">(see configure2.log)<br class=""><br class=""><br class="">Next I tried<br class="">./configure --download-superlu_dist --download-metis<br class="">--download-parmetis --download-ptscotch --with-precision=single<br class="">--with-batch<br class="">--with-mpi-dir=/mpcdf/soft/SLE_12/packages/x86_64/intel_parallel_studio/2018.4/impi/2018.4.274/intel64<br class=""><br class="">Same error message (see configure3.log)<br class=""><br class=""><br class="">It seems that there is something going on with mkl in case I want to<br class="">use Intel compilers for C and C++ (compiling with gcc and g++ seems to<br class="">work )<br class=""><br class="">Do you know what is going on there? (I am running on the Draco<br class="">cluster, if that helps<br class="">(<a href="https://www.mpcdf.mpg.de/services/computing/draco" class="">https://www.mpcdf.mpg.de/services/computing/draco</a> <<a href="https://www.mpcdf.mpg.de/services/computing/draco" class="">https://www.mpcdf.mpg.de/services/computing/draco</a>>))<br class=""><br class=""><br class="">Best regards,<br class="">Felix<br class=""><br class="">Zitat von Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a> <<a href="mailto:bsmith@petsc.dev" class="">mailto:bsmith@petsc.dev</a>>>:<br class=""><br class="">> Felix,<br class="">><br class="">> The branch is ready. Just use<br class="">><br class="">> git checkout barry/2020-07-28/superlu_dist-single<br class="">> ./configure --download-superlu_dist --download-metis<br class="">> --download-parmetis --download-ptscotch --with-precision=single<br class="">> and whatever else you use<br class="">><br class="">> Barry<br class="">><br class="">> It will automatically get the needed branch of SuperLU_DIST that<br class="">> Sherry prepared.<br class="">><br class="">><br class="">>> On Jul 27, 2020, at 2:10 PM, <a href="mailto:flw@rzg.mpg.de" class="">flw@rzg.mpg.de</a> <<a href="mailto:flw@rzg.mpg.de" class="">mailto:flw@rzg.mpg.de</a>> wrote:<br class="">>><br class="">>> Hi Shery,<br class="">>> Yes, ideally we would like to compile PETSc in single precision and<br class="">>> simply run a single precision version of SUPERLU_DIST just like<br class="">>> e.g. MUMPS.<br class="">>><br class="">>> Best regards and thanks,<br class="">>> Felix<br class="">>> Zitat von "Xiaoye S. Li" <<a href="mailto:xsli@lbl.gov" class="">xsli@lbl.gov</a> <<a href="mailto:xsli@lbl.gov" class="">mailto:xsli@lbl.gov</a>>>:<br class="">>><br class="">>>> Barry,<br class="">>>><br class="">>>> I have a branch 'Mixed-precision' working with single precision FP32. I<br class="">>>> assume Felix wants to use superlu_dist from petsc. How do you want to<br class="">>>> incorporate it in petsc?<br class="">>>><br class="">>>> <a href="https://github.com/xiaoyeli/superlu_dist" class="">https://github.com/xiaoyeli/superlu_dist</a> <<a href="https://github.com/xiaoyeli/superlu_dist" class="">https://github.com/xiaoyeli/superlu_dist</a>><br class="">>>><br class="">>>> PS1: in this version, FP32 only works on CPU. FP64 and complex-FP64 all<br class="">>>> work on GPU.<br class="">>>><br class="">>>> PS2: currently there is no mixed-precision yet, but it is the branch we are<br class="">>>> adding mix-prec support. Will take a while before merging to master.<br class="">>>><br class="">>>> Sherry<br class="">>>><br class="">>>><br class="">>>> On Wed, Jul 22, 2020 at 6:04 AM <<a href="mailto:flw@rzg.mpg.de" class="">flw@rzg.mpg.de</a> <<a href="mailto:flw@rzg.mpg.de" class="">mailto:flw@rzg.mpg.de</a>>> wrote:<br class="">>>><br class="">>>>> Hi Barry,<br class="">>>>> for now I just want to run everything in single on CPUs only with<br class="">>>>> SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but<br class="">>>>> there are no immediate plans yet. So if you could provide the support,<br class="">>>>> that would be awesome.<br class="">>>>><br class="">>>>> Best regards,<br class="">>>>> Felix<br class="">>>>><br class="">>>>> Zitat von Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a> <<a href="mailto:bsmith@petsc.dev" class="">mailto:bsmith@petsc.dev</a>>>:<br class="">>>>><br class="">>>>> > Felix,<br class="">>>>> ><br class="">>>>> > What are your needs, do you want this for CPUs or for GPUs? Do<br class="">>>>> > you wish to run all your code in single precision or just the<br class="">>>>> > SuperLU_Dist solver while the rest of your code double?<br class="">>>>> ><br class="">>>>> > If you want to run everything on CPUs using single precision<br class="">>>>> > then adding the support is very easy, we can provide that for you<br class="">>>>> > any time. The other cases will require more thought.<br class="">>>>> ><br class="">>>>> > Barry<br class="">>>>> ><br class="">>>>> ><br class="">>>>> >> On Jul 21, 2020, at 8:58 AM, <a href="mailto:flw@rzg.mpg.de" class="">flw@rzg.mpg.de</a> <<a href="mailto:flw@rzg.mpg.de" class="">mailto:flw@rzg.mpg.de</a>> wrote:<br class="">>>>> >><br class="">>>>> >> Dear PETSc support team,<br class="">>>>> >> some time ago you told me that you are planning on releasing a<br class="">>>>> >> version that supports SUPERLU_DIST in single-precision soon. Can<br class="">>>>> >> you tell me roughly what time frame you had in mind?<br class="">>>>> >><br class="">>>>> >> Best regards,<br class="">>>>> >> Felix<br class="">>>>> >><br class="">>>>><br class="">>>>><br class="">>>>><br class="">>>>><br class="">>><br class="">>><br class="">>><br class=""><br class=""><br class=""></blockquote></blockquote><br class=""><br class=""><br class=""></div></div></blockquote></div><br class=""></body></html>