<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">I submitted a job and I am waiting for the result.<br clear="all">
<div>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 21, 2020 at 3:03 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com">dave.mayhem23@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Anthony,</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 21 Jan 2020 at 08:25, Anthony Jourdon <<a href="mailto:jourdon_anthony@hotmail.fr" target="_blank">jourdon_anthony@hotmail.fr</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I made a test to try to reproduce the error.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
To do so I modified the file $PETSC_DIR/src/dm/examples/tests/ex35.c</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I attach the file in case of need.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The same error is reproduced for 1024 mpi ranks. I tested two problem sizes (2*512+1x2*64+1x2*256+1 and 2*1024+1x2*128+1x2*512+1) and the error occured for both cases, the first case is also the one I used to run before the OS and mpi updates.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I also run the code with -malloc_debug and nothing more appeared. <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I attached the configure command I used to build a debug version of petsc.</div>
</div>
</blockquote>
<div><br>
</div>
<div>The error indicates the problem occurs on the bold line below (e.g. within MPI_Isend())</div>
<div><br>
</div>
<div>
<p style="margin:0px;font:11px Menlo;color:rgb(0,132,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span> </span></span><span style="font-variant-ligatures:no-common-ligatures">/* Post the Isends with the message length-info
*/</span></p>
<p style="margin:0px;font:11px Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span>
</span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(187,44,162)">for</span><span style="font-variant-ligatures:no-common-ligatures"> (i=</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">0</span><span style="font-variant-ligatures:no-common-ligatures">,j=</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">0</span><span style="font-variant-ligatures:no-common-ligatures">;
i<size; ++i) {</span></p>
<p style="margin:0px;font:11px Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span>
</span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(187,44,162)">if</span><span style="font-variant-ligatures:no-common-ligatures"> (ilengths[i]) {</span></p>
<p style="margin:0px;font-style:normal;font-variant:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">
<b><span style="font-variant-ligatures:no-common-ligatures"><span> </span>ierr = MPI_Isend((</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(187,44,162)">void</span><span style="font-variant-ligatures:no-common-ligatures">*)(ilengths+i),</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">,MPI_INT,i,tag,comm,s_waits+j);CHKERRQ(ierr);</span></b></p>
<p style="margin:0px;font:11px Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span>
</span>j++;</span></p>
<p style="margin:0px;font:11px Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span>
</span>}</span></p>
<p style="margin:0px;font:11px Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span>
</span>}</span></p>
</div>
<div><br>
</div>
<div>The type of ilengths[i] is PetscMPIInt, which is always typedef'd to int inside PETSc.</div>
<div>I don't see how any integer mis-match (int vs long int) can be occurring so I'm puzzled what the problem is.</div>
<div>Weird...</div>
<div><br>
</div>
<div>Thanks</div>
<div>Dave</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you for your time,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sincerly.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Anthony Jourdon<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="gmail-m_5647734819507272922m_-3923578153744307772gmail-m_4324279038158345422appendonsend">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_5647734819507272922m_-3923578153744307772gmail-m_4324279038158345422divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>De :</b> Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>><br>
<b>Envoyé :</b> jeudi 16 janvier 2020 16:49<br>
<b>À :</b> Anthony Jourdon <<a href="mailto:jourdon_anthony@hotmail.fr" target="_blank">jourdon_anthony@hotmail.fr</a>><br>
<b>Cc :</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Objet :</b> Re: [petsc-users] DMDA Error</font>
<div> </div>
</div>
<div>
<div dir="ltr">It seems the problem is triggered by DMSetUp. You can write a small test creating the DMDA with the same size as your code, to see if you can reproduce the problem. If yes, it would be much easier for us to debug it.<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jan 16, 2020 at 7:38 AM Anthony Jourdon <<a href="mailto:jourdon_anthony@hotmail.fr" target="_blank">jourdon_anthony@hotmail.fr</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>
<div>
<div>
<p style="margin-top:0px;margin-bottom:0px">Dear Petsc developer,</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I need assistance with an error.<br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I run a code that uses the DMDA related functions. I'm using petsc-3.8.4.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">This code used to run very well on a super computer with the OS SLES11.</p>
<p style="margin-top:0px;margin-bottom:0px">Petsc was built using an intel mpi 5.1.3.223 module and intel mkl version 2016.0.2.181
<br>
</p>
<p style="margin-top:0px;margin-bottom:0px">The code was running with no problem on 1024 and more mpi ranks.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Recently, the OS of the computer has been updated to RHEL7<br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I rebuilt Petsc using new available versions of intel mpi (2019U5) and mkl (2019.0.5.281) which are the same versions for compilers and mkl.<br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Since then I tested to run the exact same code on 8, 16, 24, 48, 512 and 1024 mpi ranks.</p>
<p style="margin-top:0px;margin-bottom:0px">Until 1024 mpi ranks no problem, but for 1024 an error related to DMDA appeared. I snip the first lines of the error stack here and the full error stack is attached.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<div style="margin-top:0px;margin-bottom:0px">
<div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #1 PetscGatherMessageLengths() line 120 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/sys/utils/mpimesg.c</p>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #2 VecScatterCreate_PtoS() line 2288 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/vec/vec/utils/vpscat.c</p>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #3 VecScatterCreate() line 1462 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/vec/vec/utils/vscat.c</p>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #4 DMSetUp_DA_3D() line 1042 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/dm/impls/da/da3.c</p>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #5 DMSetUp_DA() line 25 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/dm/impls/da/dareg.c</p>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px">[534]PETSC ERROR: #6 DMSetUp() line 720 in /scratch2/dlp/appli_local/SCR/OROGEN/petsc3.8.4_MPI/petsc-3.8.4/src/dm/interface/dm.c</p>
</div>
</div>
<div>
<p style="margin-top:0px;margin-bottom:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px">Thank you for your time,</p>
<p style="margin-top:0px;margin-bottom:0px">Sincerly,</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Anthony Jourdon<br>
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>