<div dir="ltr">I was able to find the bug, it was the outer loop bound in the same fashion than before, my -log_view is this :<div><br></div><div><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">---------------------------------------------- PETSc Performance Summary: ----------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">./ucmsMR on a arch-linux2-c-debug named ocean with 2 processors, by valera Sat Jan 7 16:11:51 2017</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using Petsc Release Version 3.7.4, unknown </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Max Max/Min Avg Total </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Time (sec): 2.074e+01 1.00000 2.074e+01</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Objects: 9.300e+01 1.00000 9.300e+01</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Flops: 8.662e+09 1.00000 8.662e+09 1.732e+10</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Flops/sec: 4.177e+08 1.00000 4.177e+08 8.354e+08</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Memory: 1.027e+08 1.03217 2.021e+08</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MPI Messages: 5.535e+02 1.00000 5.535e+02 1.107e+03</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MPI Message Lengths: 2.533e+07 1.00000 4.576e+04 5.066e+07</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MPI Reductions: 1.903e+04 1.00000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> e.g., VecAXPY() for real vectors of length N --> 2N flops</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> and VecAXPY() for complex vectors of length N --> 8N flops</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Avg %Total Avg %Total counts %Total Avg %Total counts %Total </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> 0: Main Stage: 2.0739e+01 100.0% 1.7325e+10 100.0% 1.107e+03 100.0% 4.576e+04 100.0% 1.903e+04 100.0% </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Phase summary info:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Count: number of times phase was executed</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Time and Flops: Max - maximum over all processors</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Ratio - ratio of maximum to minimum over all processors</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mess: number of messages sent</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Avg. len: average message length (bytes)</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Reduct: number of global reductions</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Global: entire computation</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> %T - percent time in this phase %F - percent flops in this phase</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> %M - percent messages in this phase %L - percent message lengths in this phase</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> %R - percent reductions in this phase</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> ##########################################################</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # WARNING!!! #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # This code was compiled with a debugging option, #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # To get timing results run ./configure #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # using --with-debugging=no, the performance will #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # be generally two or three times faster. #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> # #</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> ##########################################################</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Event Count Time (sec) Flops --- Global --- --- Stage --- Total</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">--- Event Stage 0: Main Stage</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecDotNorm2 545 1.0 4.4925e-01 1.6 2.18e+08 1.0 0.0e+00 0.0e+00 1.1e+03 2 3 0 0 6 2 3 0 0 6 971</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecMDot 525 1.0 1.7089e+00 1.7 1.48e+09 1.0 0.0e+00 0.0e+00 1.0e+03 7 17 0 0 6 7 17 0 0 6 1735</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecNorm 420 1.0 7.6857e-02 1.0 8.40e+07 1.0 0.0e+00 0.0e+00 8.4e+02 0 1 0 0 4 0 1 0 0 4 2186</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecScale 1090 1.0 2.5113e-01 1.1 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 868</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecSet 555 1.0 7.3570e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecAXPY 1090 1.0 2.7621e-01 1.1 2.18e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1579</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecAYPX 5 1.0 3.6647e-03 2.1 5.00e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 273</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecMAXPY 1050 1.0 2.3646e+00 1.1 2.97e+09 1.0 0.0e+00 0.0e+00 0.0e+00 11 34 0 0 0 11 34 0 0 0 2508</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecAssemblyBegin 12 1.7 2.4388e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+01 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecAssemblyEnd 12 1.7 1.0085e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecScatterBegin 560 1.0 2.3770e+0071.3 0.00e+00 0.0 1.1e+03 2.7e+04 1.0e+01 6 0 99 59 0 6 0 99 59 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">VecScatterEnd 550 1.0 3.7769e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatMult 550 1.0 3.7412e+00 1.1 1.80e+09 1.0 1.1e+03 2.0e+04 0.0e+00 17 21 99 43 0 17 21 99 43 0 962</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatSolve 545 1.0 3.6138e+00 1.1 1.77e+09 1.0 0.0e+00 0.0e+00 0.0e+00 17 20 0 0 0 17 20 0 0 0 980</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatLUFactorNum 1 1.0 1.2530e-01 1.5 1.27e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 203</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatILUFactorSym 1 1.0 2.0162e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatConvert 1 1.0 3.3683e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatAssemblyBegin 1 1.0 9.5172e-02359.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatAssemblyEnd 1 1.0 2.6907e-02 1.0 0.00e+00 0.0 4.0e+00 5.0e+03 2.3e+01 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatGetRowIJ 3 1.0 1.2398e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatGetOrdering 1 1.0 4.4249e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">MatLoad 1 1.0 3.8892e-01 1.0 0.00e+00 0.0 7.0e+00 3.0e+06 3.8e+01 2 0 1 41 0 2 0 1 41 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">KSPSetUp 2 1.0 2.2634e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">KSPSolve 5 1.0 1.2218e+01 1.0 8.66e+09 1.0 1.1e+03 2.0e+04 1.9e+04 59100 99 43 99 59100 99 43 99 1418</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">PCSetUp 3 1.0 1.7993e+00 1.0 1.27e+07 1.0 0.0e+00 0.0e+00 1.0e+01 8 0 0 0 0 8 0 0 0 0 14</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">PCSetUpOnBlocks 5 1.0 1.9013e-01 1.7 1.27e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 134</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">PCApply 546 1.0 3.8320e+00 1.1 1.77e+09 1.0 0.0e+00 0.0e+00 1.0e+00 18 20 0 0 0 18 20 0 0 0 925</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Memory usage is given in bytes:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Object Type Creations Destructions Memory Descendants' Mem.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Reports information only for process 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">--- Event Stage 0: Main Stage</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Vector 72 6 5609648 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Vector Scatter 3 2 1312 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Matrix 4 0 0 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Viewer 2 0 0 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Index Set 7 4 13104 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Krylov Solver 2 0 0 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Preconditioner 3 1 1384 0.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">========================================================================================================================</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Average time to get PetscTime(): 7.15256e-08</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Average time for MPI_Barrier(): 1.82629e-05</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Average time for zero size MPI_Send(): 9.89437e-06</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">#PETSc Option Table entries:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-log_view</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-matload_block_size 1</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-pc_hypre_boomeramg_nodal_coarsen 1</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-pc_hypre_boomeramg_vec_interp_variant 1</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">#End of PETSc Option Table entries</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Compiled without FORTRAN kernels</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Compiled with full precision matrices (default)</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Configure options: --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-netcdf --download-hypre --download-metis --download-parmetis --download-trillinos</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Libraries compiled on Fri Dec 9 12:45:19 2016 on ocean </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Machine characteristics: Linux-3.10.0-327.13.1.el7.x86_64-x86_64-with-centos-7.2.1511-Core</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using PETSc directory: /home/valera/petsc</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using PETSc arch: arch-linux2-c-debug</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using C compiler: /home/valera/petsc/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g3 ${COPTFLAGS} ${CFLAGS}</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using Fortran compiler: /home/valera/petsc/arch-linux2-c-debug/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g ${FOPTFLAGS} ${FFLAGS} </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using include paths: -I/home/valera/petsc/arch-linux2-c-debug/include -I/home/valera/petsc/include -I/home/valera/petsc/include -I/home/valera/petsc/arch-linux2-c-debug/include</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using C linker: /home/valera/petsc/arch-linux2-c-debug/bin/mpicc</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using Fortran linker: /home/valera/petsc/arch-linux2-c-debug/bin/mpif90</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Using libraries: -Wl,-rpath,/home/valera/petsc/arch-linux2-c-debug/lib -L/home/valera/petsc/arch-linux2-c-debug/lib -lpetsc -Wl,-rpath,/home/valera/petsc/arch-linux2-c-debug/lib -L/home/valera/petsc/arch-linux2-c-debug/lib -lparmetis -lmetis -lHYPRE -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lmpicxx -lstdc++ -lflapack -lfblas -lnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpicxx -lstdc++ -Wl,-rpath,/home/valera/petsc/arch-linux2-c-debug/lib -L/home/valera/petsc/arch-linux2-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -ldl -Wl,-rpath,/home/valera/petsc/arch-linux2-c-debug/lib -lmpi -lgcc_s -ldl </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">WARNING! There are options you set that were not used!</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">WARNING! could be spelling mistake, etc!</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Option left: name:-pc_hypre_boomeramg_nodal_coarsen value: 1</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">Option left: name:-pc_hypre_boomeramg_vec_interp_variant value: 1</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">[valera@ocean serGCCOM]$ </span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>Any suggestions are very much appreciated, </div><div><br></div><div>Thanks</div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 3:39 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Sat, Jan 7, 2017 at 5:33 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Barry and Matt,<div><br></div><div>I was able to detect a bug that i just solved, as suggested the loop parameters weren't updated as it should, now it does and the program still freezes but now in the beginning of the loop... ?</div></div></blockquote><div><br></div></span><div>You have called a collective function from only one process. Stepping through on both processes in your run will find this easily.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Im attaching screen so you have an idea. Im thinking about it...</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 3:21 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 4:59 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I would have to think and code a MWE for this problem before sending it since the model is much bigger than the petsc solver. Attached here is a screenshot of the debugger as barry taught me, is that the stack trace you need ?<div><br></div><div>the ucmsMain.f90:522 that shows is the call (from all processes) to the routine that updates the rhs vector (from root) and scatters it (from all processes). </div></div></blockquote><div><br></div></span><div>Yes, so one process is here and the other has moved on, so there is a mismatch in calls.</div><div><br></div><div>You could do what Barry suggests, but I think it would be better to just step through your main routine once (its slow going), and</div><div>see where the divergence happens.</div><div><br></div><div> Matt</div><div><div class="m_-1595149626129250346m_-8024679799604385859h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>This routine is itself inside a double loop that occurs in all processes but the only call from all processes to the solver is this one, the rest of the loop which involves correcting for velocities, pressure and temperature, all happens in root node.</div><div><br></div><div>Sorry for the convoluted program design, this is the first beta version of the model working on parallel and was the best i could come with, i suppose it makes more sense in serial,</div><div><br></div><div>Thanks <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 2:24 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 4:20 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you Matthew, <div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 1:49 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 3:32 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Devs, hope you are having a great weekend,</div><div><br></div><div>I could finally parallelize my linear solver and implement it into the rest of the code in a way that only the linear system is solved in parallel, great news for my team, but there is a catch and is that i don't see any speedup in the linear system, i don't know if its the MPI in the cluster we are using, but im not sure on how to debug it,</div></div></blockquote><div><br></div></span><div>We need to see -log_view output for any performance question.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>On the other hand and because of this issue i was trying to do -log_summary or -log_view and i noticed the program in this context hangs when is time of producing the log, if i debug this for 2 cores, process 0 exits normally but process 1 hangs in the vectorscatterbegin() with scatter_reverse way back in the code,</div></div></blockquote><div><br></div></span><div>You are calling a collective routine from only 1 process.</div><div> <br></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div> Matt</div></div></div></div></blockquote><div><br></div><div>I am pretty confident this is not the case, </div></div></div></div></blockquote><div><br></div></span><div>This is still the simplest explanation. Can you send the stack trace for the 2 process run?</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>the callings to vecscattercreatetozero and vecscatterbegin are made in all processes, the program goes thru all of the iterations on the linear solver, writes output correctly and even closes all the petsc objects without complaining, the freeze occurs at the very end when the log is to be produced.</div></div></div></div></blockquote><div><br></div></span><div>If you can send us a code to run, we can likely find the error.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Thanks,</div><div><br></div><div>Manuel</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div> and even after destroying all associated objects and calling petscfinalize(), so im really clueless on why is this, as it only happens for -log_* or -ksp_view options.</div><div><br></div><div>my -ksp_view shows this:</div><div><br></div><div> <span style="font-family:menlo;font-size:11px">KSP Object: 2 MPI processes</span></div>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: gcr</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> GCR: restart = 30 </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> GCR: restarts performed = 20 </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> maximum iterations=10000, initial guess is zero</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerances: relative=1e-14, absolute=1e-50, divergence=10000.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> right preconditioning</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> using UNPRECONDITIONED norm type for convergence test</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">PC Object: 2 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: bjacobi</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> block Jacobi: number of blocks = 2</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Local solve is same for all blocks, in the following KSP and PC objects:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> KSP Object: (sub_) 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: preonly</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> maximum iterations=10000, initial guess is zero</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> left preconditioning</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> using NONE norm type for convergence test</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> PC Object: (sub_) 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: ilu</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> ILU: out-of-place factorization</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> 0 levels of fill</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerance for zero pivot 2.22045e-14</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> matrix ordering: natural</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> factor fill ratio given 1., needed 1.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Factored matrix follows:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: seqaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=100000, cols=100000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> package used to perform factorization: petsc</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=1675180, allocated nonzeros=1675180</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node routines</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> linear system matrix = precond matrix:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: seqaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=100000, cols=100000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=1675180, allocated nonzeros=1675180</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node routines</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> linear system matrix = precond matrix:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 2 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: mpiaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=200000, cols=200000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=3373340, allocated nonzeros=3373340</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node (on process 0) routines</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">And i configured my PC object as:</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PCSetType(mg,PCHYPRE,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PCHYPRESetType(mg,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'boomeramg'</span><span style="font-variant-ligatures:no-common-ligatures">,<wbr>ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PetscOptionsSetValue(PETSC_NUL<wbr>L_OBJECT,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'pc_hypre_boomeramg_n<wbr>odal_coarsen'</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'1'</span><span style="font-variant-ligatures:no-common-ligatures">,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PetscOptionsSetValue(PETSC_NUL<wbr>L_OBJECT,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'pc_hypre_boomeramg_v<wbr>ec_interp_variant'</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'1'</span><span style="font-variant-ligatures:no-common-ligatures">,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo">
</p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><div><br></div><div>What are your thoughts ?</div><div><br></div><div>Thanks,</div><div><br></div><div>Manuel </div><div><br></div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 6, 2017 at 1:58 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Awesome, that did it, thanks once again.<div><br></div></div><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-HOEnZb"><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 6, 2017 at 1:53 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Take the scatter out of the if () since everyone does it and get rid of the VecView().<br>
<br>
Does this work? If not where is it hanging?<br>
<span><br>
<br>
> On Jan 6, 2017, at 3:29 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
><br>
> Thanks Dave,<br>
><br>
> I think is interesting it never gave an error on this, after adding the vecassembly calls it still shows the same behavior, without complaining, i did:<br>
><br>
> if(rankl==0)then<br>
><br>
> call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
> call VecAssemblyBegin(bp0,ierr) ; call VecAssemblyEnd(bp0,ierr);<br>
> CHKERRQ(ierr)<br>
><br>
</span>endif<br>
<span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787im m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787HOEnZb">><br>
><br>
> call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
> print*,"done! "<br>
> CHKERRQ(ierr)<br>
><br>
><br>
</span><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787HOEnZb"><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787h5">> CHKERRQ(ierr)<br>
><br>
><br>
> Thanks.<br>
><br>
> On Fri, Jan 6, 2017 at 12:44 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
><br>
><br>
> On 6 January 2017 at 20:24, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> Great help Barry, i totally had overlooked that option (it is explicit in the vecscatterbegin call help page but not in vecscattercreatetozero, as i read later)<br>
><br>
> So i used that and it works partially, it scatters te values assigned in root but not the rest, if i call vecscatterbegin from outside root it hangs, the code currently look as this:<br>
><br>
> call VecScatterCreateToZero(bp2,ctr<wbr>,bp0,ierr); CHKERRQ(ierr)<br>
><br>
> call PetscObjectSetName(bp0, 'bp0:',ierr)<br>
><br>
> if(rankl==0)then<br>
><br>
> call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
><br>
> call VecView(bp0,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
><br>
><br>
> You need to call<br>
><br>
> VecAssemblyBegin(bp0);<br>
> VecAssemblyEnd(bp0);<br>
> after your last call to VecSetValues() before you can do any operations with bp0.<br>
><br>
> With your current code, the call to VecView should produce an error if you used the error checking macro CHKERRQ(ierr) (as should VecScatter{Begin,End}<br>
><br>
> Thanks,<br>
> Dave<br>
><br>
><br>
> call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
> print*,"done! "<br>
> CHKERRQ(ierr)<br>
><br>
> endif<br>
><br>
> ! call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> ! call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
><br>
> call VecView(bp2,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
><br>
> call PetscBarrier(PETSC_NULL_OBJECT<wbr>,ierr)<br>
><br>
> call exit()<br>
><br>
><br>
><br>
> And the output is: (with bp the right answer)<br>
><br>
> Vec Object:bp: 2 MPI processes<br>
> type: mpi<br>
> Process [0]<br>
> 1.<br>
> 2.<br>
> Process [1]<br>
> 4.<br>
> 3.<br>
> Vec Object:bp2: 2 MPI processes (before scatter)<br>
> type: mpi<br>
> Process [0]<br>
> 0.<br>
> 0.<br>
> Process [1]<br>
> 0.<br>
> 0.<br>
> Vec Object:bp0: 1 MPI processes<br>
> type: seq<br>
> 1.<br>
> 2.<br>
> 4.<br>
> 3.<br>
> done!<br>
> Vec Object:bp2: 2 MPI processes (after scatter)<br>
> type: mpi<br>
> Process [0]<br>
> 1.<br>
> 2.<br>
> Process [1]<br>
> 0.<br>
> 0.<br>
><br>
><br>
><br>
><br>
> Thanks inmensely for your help,<br>
><br>
> Manuel<br>
><br>
><br>
> On Thu, Jan 5, 2017 at 4:39 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> > On Jan 5, 2017, at 6:21 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> ><br>
> > Hello Devs is me again,<br>
> ><br>
> > I'm trying to distribute a vector to all called processes, the vector would be originally in root as a sequential vector and i would like to scatter it, what would the best call to do this ?<br>
> ><br>
> > I already know how to gather a distributed vector to root with VecScatterCreateToZero, this would be the inverse operation,<br>
><br>
> Use the same VecScatter object but with SCATTER_REVERSE, not you need to reverse the two vector arguments as well.<br>
><br>
><br>
> > i'm currently trying with VecScatterCreate() and as of now im doing the following:<br>
> ><br>
> ><br>
> > if(rank==0)then<br>
> ><br>
> ><br>
> > call VecCreate(PETSC_COMM_SELF,bp0,<wbr>ierr); CHKERRQ(ierr) !if i use WORLD<br>
> > !freezes in SetSizes<br>
> > call VecSetSizes(bp0,PETSC_DECIDE,n<wbr>bdp,ierr); CHKERRQ(ierr)<br>
> > call VecSetType(bp0,VECSEQ,ierr)<br>
> > call VecSetFromOptions(bp0,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
> ><br>
> > !call VecSet(bp0,5.0D0,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > call VecView(bp0,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
> ><br>
> > call VecAssemblyBegin(bp0,ierr) ; call VecAssemblyEnd(bp0,ierr) !rhs<br>
> ><br>
> > do i=0,nbdp-1,1<br>
> > ind(i+1) = i<br>
> > enddo<br>
> ><br>
> > call ISCreateGeneral(PETSC_COMM_SEL<wbr>F,nbdp,ind,PETSC_COPY_VALUES,l<wbr>ocis,ierr)<br>
> ><br>
> > !call VecScatterCreate(bp0,PETSC_NUL<wbr>L_OBJECT,bp2,is,ctr,ierr) !if i use SELF<br>
> > !freezes here.<br>
> ><br>
> > call VecScatterCreate(bp0,locis,bp2<wbr>,PETSC_NULL_OBJECT,ctr,ierr)<br>
> ><br>
> > endif<br>
> ><br>
> > bp2 being the receptor MPI vector to scatter to<br>
> ><br>
> > But it freezes in VecScatterCreate when trying to use more than one processor, what would be a better approach ?<br>
> ><br>
> ><br>
> > Thanks once again,<br>
> ><br>
> > Manuel<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Wed, Jan 4, 2017 at 3:30 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks i had no idea how to debug and read those logs, that solved this issue at least (i was sending a message from root to everyone else, but trying to catch from everyone else including root)<br>
> ><br>
> > Until next time, many thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 3:23 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> > On Wed, Jan 4, 2017 at 5:21 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > I did a PetscBarrier just before calling the vicariate routine and im pretty sure im calling it from every processor, code looks like this:<br>
> ><br>
> > From the gdb trace.<br>
> ><br>
> > Proc 0: Is in some MPI routine you call yourself, line 113<br>
> ><br>
> > Proc 1: Is in VecCreate(), line 130<br>
> ><br>
> > You need to fix your communication code.<br>
> ><br>
> > Matt<br>
> ><br>
> > call PetscBarrier(PETSC_NULL_OBJECT<wbr>,ierr)<br>
> ><br>
> > print*,'entering POInit from',rank<br>
> > !call exit()<br>
> ><br>
> > call PetscObjsInit()<br>
> ><br>
> ><br>
> > And output gives:<br>
> ><br>
> > entering POInit from 0<br>
> > entering POInit from 1<br>
> > entering POInit from 2<br>
> > entering POInit from 3<br>
> ><br>
> ><br>
> > Still hangs in the same way,<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:55 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks for the answers !<br>
> ><br>
> > heres the screenshot of what i got from bt in gdb (great hint in how to debug in petsc, didn't know that)<br>
> ><br>
> > I don't really know what to look at here,<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:39 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> > Are you certain ALL ranks in PETSC_COMM_WORLD call these function(s). These functions cannot be inside if statements like<br>
> > if (rank == 0){<br>
> > VecCreateMPI(...)<br>
> > }<br>
> ><br>
> ><br>
> > On Wed, 4 Jan 2017 at 23:34, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks Dave for the quick answer, appreciate it,<br>
> ><br>
> > I just tried that and it didn't make a difference, any other suggestions ?<br>
> ><br>
> > Thanks,<br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:29 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> > You need to swap the order of your function calls.<br>
> > Call VecSetSizes() before VecSetType()<br>
> ><br>
> > Thanks,<br>
> > Dave<br>
> ><br>
> ><br>
> > On Wed, 4 Jan 2017 at 23:21, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Hello all, happy new year,<br>
> ><br>
> > I'm working on parallelizing my code, it worked and provided some results when i just called more than one processor, but created artifacts because i didn't need one image of the whole program in each processor, conflicting with each other.<br>
> ><br>
> > Since the pressure solver is the main part i need in parallel im chosing mpi to run everything in root processor until its time to solve for pressure, at this point im trying to create a distributed vector using either<br>
> ><br>
> > call VecCreateMPI(PETSC_COMM_WORLD,<wbr>PETSC_DECIDE,nbdp,xp,ierr)<br>
> > or<br>
> > call VecCreate(PETSC_COMM_WORLD,xp,<wbr>ierr); CHKERRQ(ierr)<br>
> > call VecSetType(xp,VECMPI,ierr)<br>
> > call VecSetSizes(xp,PETSC_DECIDE,nb<wbr>dp,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > In both cases program hangs at this point, something it never happened on the naive way i described before. I've made sure the global size, nbdp, is the same in every processor. What can be wrong?<br>
> ><br>
> > Thanks for your kind help,<br>
> ><br>
> > Manuel.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
</blockquote></div></div></div><div><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124h5"><br><br clear="all"><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></font></span></div></div></div></div><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><br></font></span></div></div><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><div><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091h5"><br><br clear="all"><span class="m_-1595149626129250346HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div></font></span></div></div><span class="m_-1595149626129250346HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="m_-1595149626129250346HOEnZb"><font color="#888888"><br></font></span></div></div><span class="m_-1595149626129250346HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="m_-1595149626129250346HOEnZb"><font color="#888888"><div><div class="m_-1595149626129250346m_-8024679799604385859h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-1595149626129250346m_-8024679799604385859m_-3650215258836467221gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-1595149626129250346gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div></div></div>
</blockquote></div><br></div>