<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On May 29, 2018, at 3:05 AM, Najeeb Ahmad <<a href="mailto:nahmad16@ku.edu.tr" class="">nahmad16@ku.edu.tr</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><br class="Apple-interchange-newline">
<br style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
On Mon, May 28, 2018 at 9:32 PM, Smith, Barry F.<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:bsmith@mcs.anl.gov" target="_blank" class="">bsmith@mcs.anl.gov</a>></span><span class="Apple-converted-space"> </span>wrote:<br class="">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<span class=""><br class="">
<br class="">
> On May 28, 2018, at 10:32 AM, Najeeb Ahmad <<a href="mailto:nahmad16@ku.edu.tr" class="">nahmad16@ku.edu.tr</a>> wrote:<br class="">
><span class="Apple-converted-space"> </span><br class="">
> Thanks a lot Satish for your prompt reply.<br class="">
><span class="Apple-converted-space"> </span><br class="">
> I just checked that SuperLU_dist package works only for matrices of type aij. and uses lu preconditioner. I am currently working with baij matrix. What is the best preconditioner choice for baij matrices on parallel machines?<br class="">
<br class="">
</span> The best preconditioner is always problem specific. Where does your problem come from? CFD? Structural mechanics? other apps?<br class="">
</blockquote>
<div class=""> </div>
<div class=""> <span class="Apple-converted-space"> </span><font color="#0000ff" class="">I am interested in writing solver for reservoir simulation employing FVM and unstructured grids. My main objective is to study performance of the code with different
data structures/data layouts and architecture specific optimizations, specifically targeting the multicore architectures like KNL for instance. Later the study may be extended to include GPUs. The options for switching between AIJ and BAIJ etc. are therefore
very useful for my study. </font></div>
<div class=""><font color="#0000ff" class=""><br class="">
</font></div>
<div class=""><font color="#0000ff" class=""> The purpose why I wanted to change the preconditioner is that the default preconditioner is giving me different iterations count for different number of processesors. I would rather like a preconditioner
that would give me same iteration count for any processor count so that I can better compare the performance results. </font></div>
<div class=""><font color="#0000ff" class=""><br class="">
</font></div>
<div class=""><font color="#0000ff" class=""> Your suggestions in this regard are highly appreciated, specifically with reference to the following points:</font></div>
<div class=""><font color="#0000ff" class=""> </font></div>
<div class=""><font color="#0000ff" class=""> - Is it possible to explicitly use high bandwidth memory in PETSc for selected object placement (e.g. using memkind library for instance)?</font></div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Yes, see <a href="http://www.mcs.anl.gov/petsc/petsc-3.8/src/sys/memory/mhbw.c.html" class="">http://www.mcs.anl.gov/petsc/petsc-3.8/src/sys/memory/mhbw.c.html</a></div>
<div><br class="">
</div>
<div>This was developed to use memkind to handle adjoint checkpointing where I want to use HBW memory for computation and DRAM for storing checkpoints. But it can be used for your purpose as well.</div>
<div><br class="">
</div>
<div>When configure PETSc, use "--with-memkind-dir=" to specify the location of the memkind library.</div>
<div>The runtime option "-malloc_hbw" will allow you to allocate all PETSc objects in HBW memory. If the HBW memory is ran out, it falls back to DRAM.</div>
<div><br class="">
</div>
<div>If you want to place selective objects in DRAM, you can do</div>
<div><br class="">
</div>
<div>PetscMallocSetDRAM() </div>
<div>... allocate your objects ...</div>
<div><span style="orphans: 2; white-space: pre-wrap; widows: 2;" class="">PetscMallocResetDRAM()</span></div>
<div class=""><br class="">
</div>
<div class="">An example usage can be found at</div>
<div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-dev/src/ts/trajectory/impls/memory/trajmemory.c" class="">http://www.mcs.anl.gov/petsc/petsc-dev/src/ts/trajectory/impls/memory/trajmemory.c</a></div>
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class=""><font color="#0000ff" class=""> - What would it take to take advantage of architecture specific compiler flags to achieve good performance on a given platform (e.g. -xMIC-AVX512 for AVX512 on KNL, #pragma SIMD etc.). </font></div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>To build PETSc on KNL with AVX512 enabled, see the example scripts </div>
<div>config/examples/arch-linux-knl.py</div>
<div>config/examples/arch-cray-xc40-knl-opt.py</div>
<div><br class="">
</div>
<div>Note that the MatMult kernel (for AIJ and SELL) has been manually optimized for best performance.</div>
<div><br class="">
</div>
<div>Hong (Mr.)</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class=""><font color="#0000ff" class=""> Sorry for some very basic questions as I am a novice PETSc user. </font></div>
<div class=""><font color="#0000ff" class=""><br class="">
</font></div>
<div class=""><font color="#0000ff" class=""> Thanks for your time :)</font></div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<br class="">
Anyways you probably want to make your code be able to switch between AIJ and BAIJ at run time since the different formats support somewhat different solvers. If your code alls MatSetFromOptions then you can switch via the command line option -mat_type
aij or baij <span class="Apple-converted-space"> </span><br class="">
<span class="HOEnZb"><font color="#888888" class=""><br class="">
Barry<br class="">
</font></span>
<div class="HOEnZb">
<div class="h5"><br class="">
><span class="Apple-converted-space"> </span><br class="">
> Thanks<br class="">
><span class="Apple-converted-space"> </span><br class="">
> On Mon, May 28, 2018 at 8:23 PM, Satish Balay <<a href="mailto:balay@mcs.anl.gov" class="">balay@mcs.anl.gov</a>> wrote:<br class="">
> On Mon, 28 May 2018, Najeeb Ahmad wrote:<br class="">
><span class="Apple-converted-space"> </span><br class="">
> > Hi All,<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > I have Petsc release version 3.9.2 configured with the following options:<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort<br class="">
> > --download-fblaslapack=1<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > Now I want to use PCILU in my code and when I set the PC type to PCILU in<br class="">
> > the code, I get the following error:<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > [0]PETSC ERROR: --------------------- Error Message<br class="">
> > ------------------------------<wbr class="">------------------------------<wbr class="">--<br class="">
> > [0]PETSC ERROR: See<br class="">
> ><span class="Apple-converted-space"> </span><a href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html" rel="noreferrer" target="_blank" class="">http://www.mcs.anl.gov/petsc/<wbr class="">documentation/<wbr class="">linearsolvertable.html</a><span class="Apple-converted-space"> </span>for<br class="">
> > possible LU and Cholesky solvers<br class="">
> > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must<br class="">
> > ./configure with --download-<package><br class="">
> > [0]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank" class="">http://www.mcs.anl.gov/petsc/<wbr class="">documentation/faq.html</a><span class="Apple-converted-space"> </span>for<br class="">
> > trouble shooting.<br class="">
> > [0]PETSC ERROR: Petsc Release Version 3.9.2, unknown<br class="">
> > [0]PETSC ERROR: ./main on a arch-linux2-c-debug named Karachi by nahmad Mon<br class="">
> > May 28 17:52:41 2018<br class="">
> > [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc<br class="">
> > --with-fc=mpiifort --download-fblaslapack=1<br class="">
> > [0]PETSC ERROR: #1 MatGetFactor() line 4318 in<br class="">
> > /home/nahmad/PETSc/petsc/src/<wbr class="">mat/interface/matrix.c<br class="">
> > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in<br class="">
> > /home/nahmad/PETSc/petsc/src/<wbr class="">ksp/pc/impls/factor/ilu/ilu.c<br class="">
> > [0]PETSC ERROR: #3 PCSetUp() line 923 in<br class="">
> > /home/nahmad/PETSc/petsc/src/<wbr class="">ksp/pc/interface/precon.c<br class="">
> > [0]PETSC ERROR: #4 KSPSetUp() line 381 in<br class="">
> > /home/nahmad/PETSc/petsc/src/<wbr class="">ksp/ksp/interface/itfunc.c<br class="">
> > [0]PETSC ERROR: #5 KSPSolve() line 612 in<br class="">
> > /home/nahmad/PETSc/petsc/src/<wbr class="">ksp/ksp/interface/itfunc.c<br class="">
> > [0]PETSC ERROR: #6 SolveSystem() line 60 in<br class="">
> > /home/nahmad/Aramco/petsc/<wbr class="">petsc/BlockSolveTest/src/main.<wbr class="">c<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > I assume that I am missing LU package like SuperLU_dist for instance and I<br class="">
> > need to download and configure it with Petsc.<br class="">
><span class="Apple-converted-space"> </span><br class="">
> yes - petsc has sequential LU - but you need superlu_dist/mumps for parallel lu.<br class="">
><span class="Apple-converted-space"> </span><br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > I am wondering what is the best way to reconfigure Petsc to download and<br class="">
> > use the appropriate package to support PCILU?<br class="">
><span class="Apple-converted-space"> </span><br class="">
> Rerun configure with the additional option --download-superlu_dist=1.<br class="">
><span class="Apple-converted-space"> </span><br class="">
> You can do this with current PETSC_ARCH you are using [i.e reinstall<br class="">
> over the current build] - or use a different PETSC_ARCH - so both<br class="">
> builds exist and useable.<br class="">
><span class="Apple-converted-space"> </span><br class="">
> Satish<br class="">
><span class="Apple-converted-space"> </span><br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> > You advice is highly appreciated.<br class="">
> ><span class="Apple-converted-space"> </span><br class="">
> ><span class="Apple-converted-space"> </span><br class="">
><span class="Apple-converted-space"> </span><br class="">
><span class="Apple-converted-space"> </span><br class="">
><span class="Apple-converted-space"> </span><br class="">
><span class="Apple-converted-space"> </span><br class="">
> --<span class="Apple-converted-space"> </span><br class="">
> Najeeb Ahmad<br class="">
><span class="Apple-converted-space"> </span><br class="">
> Research and Teaching Assistant<br class="">
> PARallel and MultiCORE Computing Laboratory (ParCoreLab)<span class="Apple-converted-space"> </span><br class="">
> Computer Science and Engineering<br class="">
> Koç University, Istanbul, Turkey<br class="">
><span class="Apple-converted-space"> </span><br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br clear="all" style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_signature" data-smartmail="gmail_signature" style="caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class=""></div>
<font size="2" class=""><b class=""><span style="color: rgb(7, 55, 99);" class=""><i class="">Najeeb Ahmad</i></span></b></font></div>
<div class=""><font size="2" class=""><b class=""><font color="#073763" class=""><i class=""><br class="">
</i></font></b></font>
<div dir="ltr" style="font-size: small;" class="">
<div class=""><font size="2" class=""><b class=""><span style="color: rgb(120, 63, 4);" class="">Research and Teaching Assistant</span><br class="">
</b></font></div>
<div class=""><font size="2" class=""><b class=""><font class="">PARallel and MultiCORE Computing Laboratory (ParCoreLab)</font><span style="color: rgb(120, 63, 4);" class=""> </span></b></font></div>
<font size="2" class=""><b class="">Computer Science and Engineering<br class="">
</b></font></div>
<div style="font-size: small;" class=""><font size="2" class=""><b class="">Koç University, Istanbul, Turkey</b></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>