<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Thanks for the fast reply here I attached the log_view from my program on 96 processors and 1536 processors vic_output_96 and vic_output_1536. I also attached the output of ex34 increased in a grid 1600x512x512. The good news is that in the example 34 it
seem to scale at least looking at KSPSolve function time. What I do not understand are all the other numbers in the profiler. In exercise 34 on 96 processor VecScale and VecNormalize these call that take most of the time 148 and 153 seconds (that for me is
already surprising) on 1536 these call suddenly drop to 0.1s and 2.2 seconds that are higher than a factor x16.</p>
<p><br>
</p>
<p>I will try now other examples more in the direction of having only CG and no preconditioners to see what happen in scalability and understand what I am doing wrong. But in the meanwhile I have a second question does the fact that I compiled PETSC with debugging
= 0 could affect the profiler numbers to be unreliable ?</p>
<p><br>
</p>
<p>Thanks in advance</p>
<p>Pietro Incardona<br>
</p>
<p><br>
</p>
<p><br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>Da:</b> Barry Smith <bsmith@mcs.anl.gov><br>
<b>Inviato:</b> giovedì 15 giugno 2017 23:16:50<br>
<b>A:</b> Pietro Incardona<br>
<b>Cc:</b> petsc-users@mcs.anl.gov<br>
<b>Oggetto:</b> Re: [petsc-users] PETSC profiling on 1536 cores</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
Please send the complete -log_view files as attachments. Both 1536 and 48 cores. The mailers mess up the ASCII formatting in text so I can't make heads or tails out of the result.<br>
<br>
What is the MPI being used and what kind of interconnect does the network have? Also is the MPI specific to that interconnect or just something compiled off the web?<br>
<br>
<br>
Barry<br>
<br>
> On Jun 15, 2017, at 4:09 PM, Pietro Incardona <asmprog32@hotmail.com> wrote:<br>
> <br>
> Dear All<br>
> <br>
> I tried PETSC version 3.6.5 to solve a linear system with 256 000 000 unknown. The equation is Finite differences Poisson equation.<br>
> <br>
> I am using Conjugate gradient (the matrix is symmetric) with no preconditioner. Visualizing the solution is reasonable.<br>
> Unfortunately the Conjugate-Gradient does not scale at all and I am extremely concerned about this problem in paticular about the profiling numbers.
<br>
> Looking at the profiler it seem that<br>
> <br>
> 1536 cores = 24 cores x 64<br>
> <br>
> VecScatterBegin 348 1.0 2.3975e-01 1.8 0.00e+00 0.0 7.7e+06 3.1e+04 0.0e+00 0 0 85 99 0 0 0 85 99 0 0<br>
> VecScatterEnd 348 1.0 2.8680e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
> MatMult 348 1.0 4.1088e+00 1.4 8.18e+08 1.3 7.7e+06 3.1e+04 0.0e+00 2 52 85 99 0 2 52 85 99 0 281866<br>
> <br>
> I was expecting that this part was the most expensive and take around 4 second in total that sound reasonable<br>
> <br>
> Unfortunately<br>
> <br>
> on 1536 cores = 24 cores x 64<br>
> <br>
> VecTDot 696 1.0 3.4442e+01 1.4 2.52e+08 1.3 0.0e+00 0.0e+00 7.0e+02 12 16 0 0 65 12 16 0 0 65 10346<br>
> VecNorm 349 1.0 1.1101e+02 1.1 1.26e+08 1.3 0.0e+00 0.0e+00 3.5e+02 46 8 0 0 33 46 8 0 0 33 1610<br>
> VecAXPY 696 1.0 8.3134e+01 1.1 2.52e+08 1.3 0.0e+00 0.0e+00 0.0e+00 34 16 0 0 0 34 16 0 0 0 4286<br>
> <br>
> Take over 228 seconds. Considering that doing some test on the cluster I do not see any problem with MPI_Reduce I do not understand how these numbers are possible<br>
> <br>
> <br>
> ////////////////////////// I also attach to the profiling part the inversion on 48 cores /////////////////////////<br>
> <br>
> VecTDot 696 1.0 1.4684e+01 1.3 3.92e+09 1.1 0.0e+00 0.0e+00 7.0e+02 6 16 0 0 65 6 16 0 0 65 24269<br>
> VecNorm 349 1.0 4.9612e+01 1.3 1.96e+09 1.1 0.0e+00 0.0e+00 3.5e+02 22 8 0 0 33 22 8 0 0 33 3602<br>
> VecCopy 351 1.0 8.8359e+00 7.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
> VecSet 2 1.0 1.6177e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecAXPY 696 1.0 8.8559e+01 1.1 3.92e+09 1.1 0.0e+00 0.0e+00 0.0e+00 42 16 0 0 0 42 16 0 0 0 4024<br>
> VecAYPX 347 1.0 4.6790e+00 1.2 1.95e+09 1.1 0.0e+00 0.0e+00 0.0e+00 2 8 0 0 0 2 8 0 0 0 37970<br>
> VecAssemblyBegin 2 1.0 5.0942e-02 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 1 0 0 0 0 1 0<br>
> VecAssemblyEnd 2 1.0 1.9073e-05 6.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecScatterBegin 348 1.0 1.2763e+00 1.5 0.00e+00 0.0 4.6e+05 2.0e+05 0.0e+00 0 0 97100 0 0 0 97100 0 0<br>
> VecScatterEnd 348 1.0 4.6741e+00 5.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
> MatMult 348 1.0 2.8440e+01 1.1 1.27e+10 1.1 4.6e+05 2.0e+05 0.0e+00 13 52 97100 0 13 52 97100 0 40722<br>
> MatAssemblyBegin 1 1.0 7.4749e-0124.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatAssemblyEnd 1 1.0 8.3194e-01 1.0 0.00e+00 0.0 2.7e+03 5.1e+04 8.0e+00 0 0 1 0 1 0 0 1 0 1 0<br>
> KSPSetUp 1 1.0 8.2883e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> KSPSolve 1 1.0 1.7964e+02 1.0 2.45e+10 1.1 4.6e+05 2.0e+05 1.0e+03 87100 97100 98 87100 97100 98 12398<br>
> PCSetUp 1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> PCApply 349 1.0 8.8166e+00 7.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
> <br>
> <br>
> /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////<br>
> <br>
> If you need more information or test please let me know.<br>
> <br>
> Thanks in advance<br>
> <br>
> Here the log of 1536 cores<br>
> <br>
> 345 KSP Residual norm 1.007085286893e-02 <br>
> 346 KSP Residual norm 1.010054402040e-02 <br>
> 347 KSP Residual norm 1.002139574355e-02 <br>
> 348 KSP Residual norm 9.775851299055e-03 <br>
> Max div for vorticity 1.84572e-05 Integral: 6.62466e-09 130.764 -132<br>
> ************************************************************************************************************************<br>
> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***<br>
> ************************************************************************************************************************<br>
> <br>
> ---------------------------------------------- PETSc Performance Summary: ----------------------------------------------<br>
> <br>
> ./vic_petsc on a arch-linux2-c-opt named taurusi6217 with 1536 processors, by incard Thu Jun 15 22:27:09 2017<br>
> Using Petsc Release Version 3.6.4, Apr, 12, 2016 <br>
> <br>
> Max Max/Min Avg Total <br>
> Time (sec): 2.312e+02 1.00027 2.312e+02<br>
> Objects: 1.900e+01 1.00000 1.900e+01<br>
> Flops: 1.573e+09 1.32212 1.450e+09 2.227e+12<br>
> Flops/sec: 6.804e+06 1.32242 6.271e+06 9.633e+09<br>
> MPI Messages: 8.202e+03 2.06821 5.871e+03 9.018e+06<br>
> MPI Message Lengths: 2.013e+08 1.86665 2.640e+04 2.381e+11<br>
> MPI Reductions: 1.066e+03 1.00000<br>
> <br>
> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)<br>
> e.g., VecAXPY() for real vectors of length N --> 2N flops<br>
> and VecAXPY() for complex vectors of length N --> 8N flops<br>
> <br>
> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --<br>
> Avg %Total Avg %Total counts %Total Avg %Total counts %Total
<br>
> 0: Main Stage: 2.3120e+02 100.0% 2.2272e+12 100.0% 9.018e+06 100.0% 2.640e+04 100.0% 1.065e+03 99.9%
<br>
> <br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> See the 'Profiling' chapter of the users' manual for details on interpreting output.<br>
> Phase summary info:<br>
> Count: number of times phase was executed<br>
> Time and Flops: Max - maximum over all processors<br>
> Ratio - ratio of maximum to minimum over all processors<br>
> Mess: number of messages sent<br>
> Avg. len: average message length (bytes)<br>
> Reduct: number of global reductions<br>
> Global: entire computation<br>
> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().<br>
> %T - percent time in this phase %F - percent flops in this phase<br>
> %M - percent messages in this phase %L - percent message lengths in this phase<br>
> %R - percent reductions in this phase<br>
> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> Event Count Time (sec) Flops --- Global --- --- Stage --- Total<br>
> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> <br>
> --- Event Stage 0: Main Stage<br>
> <br>
> VecTDot 696 1.0 3.4442e+01 1.4 2.52e+08 1.3 0.0e+00 0.0e+00 7.0e+02 12 16 0 0 65 12 16 0 0 65 10346<br>
> VecNorm 349 1.0 1.1101e+02 1.1 1.26e+08 1.3 0.0e+00 0.0e+00 3.5e+02 46 8 0 0 33 46 8 0 0 33 1610<br>
> VecCopy 351 1.0 2.7609e-01 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecSet 2 1.0 3.8961e-0256.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecAXPY 696 1.0 8.3134e+01 1.1 2.52e+08 1.3 0.0e+00 0.0e+00 0.0e+00 34 16 0 0 0 34 16 0 0 0 4286<br>
> VecAYPX 347 1.0 2.0852e-01 2.0 1.25e+08 1.3 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 8 0 0 0 852044<br>
> VecAssemblyBegin 2 1.0 8.3237e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 1 0 0 0 0 1 0<br>
> VecAssemblyEnd 2 1.0 5.1022e-0517.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecScatterBegin 348 1.0 2.3975e-01 1.8 0.00e+00 0.0 7.7e+06 3.1e+04 0.0e+00 0 0 85 99 0 0 0 85 99 0 0<br>
> VecScatterEnd 348 1.0 2.8680e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
> MatMult 348 1.0 4.1088e+00 1.4 8.18e+08 1.3 7.7e+06 3.1e+04 0.0e+00 2 52 85 99 0 2 52 85 99 0 281866<br>
> MatAssemblyBegin 1 1.0 9.1920e-02 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatAssemblyEnd 1 1.0 3.9093e+00 1.0 0.00e+00 0.0 4.4e+04 7.7e+03 8.0e+00 2 0 0 0 1 2 0 0 0 1 0<br>
> KSPSetUp 1 1.0 8.1890e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> KSPSolve 1 1.0 2.1525e+02 1.0 1.57e+09 1.3 7.7e+06 3.1e+04 1.0e+03 93100 85 99 98 93100 85 99 98 10347<br>
> PCSetUp 1 1.0 5.5075e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> PCApply 349 1.0 2.7485e-01 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> <br>
> Memory usage is given in bytes:<br>
> <br>
> Object Type Creations Destructions Memory Descendants' Mem.<br>
> Reports information only for process 0.<br>
> <br>
> --- Event Stage 0: Main Stage<br>
> <br>
> Vector 10 7 7599552 0<br>
> Vector Scatter 1 1 1088 0<br>
> Matrix 3 3 20858912 0<br>
> Krylov Solver 1 1 1216 0<br>
> Index Set 2 2 242288 0<br>
> Preconditioner 1 1 816 0<br>
> Viewer 1 0 0 0<br>
> ========================================================================================================================<br>
> Average time to get PetscTime(): 9.53674e-08<br>
> Average time for MPI_Barrier(): 3.68118e-05<br>
> Average time for zero size MPI_Send(): 3.24349e-06<br>
> #PETSc Option Table entries:<br>
> -ksp_atol 0.010000<br>
> -ksp_max_it 500<br>
> -ksp_monitor<br>
> -ksp_type cg<br>
> -log_summary<br>
> #End of PETSc Option Table entries<br>
> Compiled without FORTRAN kernels<br>
> Compiled with full precision matrices (default)<br>
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4<br>
> Configure options: --with-cxx-dialect=C++11 --with-mpi-dir=/sw/taurus/libraries/openmpi/1.10.2-gnu --with-parmetis=yes --with-parmetis-dir=/scratch/p_ppm//PARMETIS/ --with-metis=yes --with-metis-dir=/scratch/p_ppm//METIS --with-boost=yes --with-boost-dir=/scratch/p_ppm//BOOST
--with-blas-lib=/scratch/p_ppm//OPENBLAS/lib/libopenblas.a --with-lapack-lib=/scratch/p_ppm//OPENBLAS/lib/libopenblas.a --with-suitesparse=yes --with-suitesparse-dir=/scratch/p_ppm//SUITESPARSE --with-trilinos=yes -with-trilinos-dir=/scratch/p_ppm//TRILINOS
--with-scalapack=yes -with-scalapack-dir=/scratch/p_ppm//SCALAPACK --with-mumps=yes --with-mumps-include=/scratch/p_ppm//MUMPS/include --with-superlu_dist=yes --with-superlu_dist-lib=/scratch/p_ppm//SUPERLU_DIST/lib/libsuperlu_dist_4.3.a --with-superlu_dist-include=/scratch/p_ppm//SUPERLU_DIST/include/
--with-hypre=yes -with-hypre-dir=/scratch/p_ppm//HYPRE --with-mumps-lib=""/scratch/p_ppm//MUMPS/lib/libdmumps.a /scratch/p_ppm//MUMPS/lib/libmumps_common.a /scratch/p_ppm//MUMPS/lib/libpord.a"" --prefix=/scratch/p_ppm//PETSC --with-debugging=0<br>
> -----------------------------------------<br>
> Libraries compiled on Wed Feb 22 17:30:49 2017 on tauruslogin4 <br>
> Machine characteristics: Linux-2.6.32-642.11.1.el6.Bull.106.x86_64-x86_64-with-redhat-6.8-Santiago<br>
> Using PETSc directory: /lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4<br>
> Using PETSc arch: arch-linux2-c-opt<br>
> -----------------------------------------<br>
> <br>
> Using C compiler: /sw/taurus/libraries/openmpi/1.10.2-gnu/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS}<br>
> Using Fortran compiler: /sw/taurus/libraries/openmpi/1.10.2-gnu/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS}
<br>
> -----------------------------------------<br>
> <br>
> Using include paths: -I/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/arch-linux2-c-opt/include -I/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/include -I/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/include
-I/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/arch-linux2-c-opt/include -I/scratch/p_ppm/TRILINOS/include -I/scratch/p_ppm/HYPRE/include -I/scratch/p_ppm/SUPERLU_DIST/include -I/scratch/p_ppm/SUITESPARSE/include -I/scratch/p_ppm/MUMPS/include
-I/scratch/p_ppm/PARMETIS/include -I/scratch/p_ppm/METIS/include -I/scratch/p_ppm/BOOST/include -I/sw/taurus/libraries/openmpi/1.10.2-gnu/include<br>
> -----------------------------------------<br>
> <br>
> Using C linker: /sw/taurus/libraries/openmpi/1.10.2-gnu/bin/mpicc<br>
> Using Fortran linker: /sw/taurus/libraries/openmpi/1.10.2-gnu/bin/mpif90<br>
> Using libraries: -Wl,-rpath,/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/arch-linux2-c-opt/lib -L/lustre/scratch2/p_ppm/jenkins2/workspace/OpenFPM_high_scal_tests/petsc-3.6.4/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/scratch/p_ppm/TRILINOS/lib
-L/scratch/p_ppm/TRILINOS/lib -lpike-blackbox -ltrilinoscouplings -lmsqutil -lmesquite -lctrilinos -lsundancePdeopt -lsundanceStdFwk -lsundanceStdMesh -lsundanceCore -lsundanceInterop -lsundanceUtils -lsundancePlaya -lpiro -lrol -lstokhos_muelu -lstokhos_ifpack2
-lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lmoochothyra -lmoocho -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -lmoertel -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_mesh_fixtures -lstk_search_util_base
-lstk_search -lstk_unit_test_utils -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_diag -lstk_util_env -lstk_util_util -lstkclassic_search_util -lstkclassic_search -lstkclassic_rebalance_utils -lstkclassic_rebalance
-lstkclassic_linsys -lstkclassic_io_util -lstkclassic_io -lstkclassic_expreval -lstkclassic_algsup -lstkclassic_mesh_fem -lstkclassic_mesh_base -lstkclassic_util_use_cases -lstkclassic_util_unit_test_support -lstkclassic_util_parallel -lstkclassic_util_diag
-lstkclassic_util_env -lstkclassic_util_util -lstk_mesh_fixtures -lstk_search_util_base -lstk_search -lstk_unit_test_utils -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_diag -lstk_util_env -lstk_util_util
-lstkclassic_search_util -lstkclassic_search -lstkclassic_rebalance_utils -lstkclassic_rebalance -lstkclassic_linsys -lstkclassic_io_util -lstkclassic_io -lstkclassic_expreval -lstkclassic_algsup -lstkclassic_mesh_fem -lstkclassic_mesh_base -lstkclassic_util_use_cases
-lstkclassic_util_unit_test_support -lstkclassic_util_parallel -lstkclassic_util_diag -lstkclassic_util_env -lstkclassic_util_util -lintrepid -lteko -lfei_trilinos -lfei_base -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml
-lstratimikosifpack -lifpack2-adapters -lifpack2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lkomplex -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoss -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoss
-lamesos2 -lshylu -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lzoltan2 -lpamgen_extras -lpamgen -lamesos -lgaleri-xpetra -lgaleri-epetra -laztecoo -ldpliris -lisorropia -loptipack -lxpetra-sup -lxpetra -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore
-lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels
-ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore
-lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest -lpthread
-Wl,-rpath,/scratch/p_ppm/HYPRE/lib -L/scratch/p_ppm/HYPRE/lib -lHYPRE -Wl,-rpath,/sw/taurus/libraries/openmpi/1.10.2-gnu/lib -L/sw/taurus/libraries/openmpi/1.10.2-gnu/lib -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib64 -L/sw/global/compilers/gcc/5.3.0/lib64
-Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/libexec/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/sw/global/compilers/gcc/5.3.0/libexec/gcc/x86_64-unknown-linux-gnu/5.3.0
-Wl,-rpath,/sw/global/compilers/gcc/5.3.0/x86_64-unknown-linux-gnu/lib -L/sw/global/compilers/gcc/5.3.0/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib -L/sw/global/compilers/gcc/5.3.0/lib -lmpi_cxx -lstdc++ -Wl,-rpath,/scratch/p_ppm//SUPERLU_DIST/lib
-L/scratch/p_ppm//SUPERLU_DIST/lib -lsuperlu_dist_4.3 -Wl,-rpath,/scratch/p_ppm/SUITESPARSE/lib -L/scratch/p_ppm/SUITESPARSE/lib -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lrt -ldmumps -Wl,-rpath,/scratch/p_ppm//MUMPS/lib
-L/scratch/p_ppm//MUMPS/lib -lmumps_common -lpord -Wl,-rpath,/scratch/p_ppm/SCALAPACK/lib -L/scratch/p_ppm/SCALAPACK/lib -lscalapack -Wl,-rpath,/scratch/p_ppm//OPENBLAS/lib -L/scratch/p_ppm//OPENBLAS/lib -lopenblas -Wl,-rpath,/scratch/p_ppm/PARMETIS/lib -L/scratch/p_ppm/PARMETIS/lib
-lparmetis -Wl,-rpath,/scratch/p_ppm/METIS/lib -L/scratch/p_ppm/METIS/lib -lmetis -lX11 -lhwloc -lssl -lcrypto -lm -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/sw/taurus/libraries/openmpi/1.10.2-gnu/lib
-L/sw/taurus/libraries/openmpi/1.10.2-gnu/lib -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib64 -L/sw/global/compilers/gcc/5.3.0/lib64 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0
-Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib64 -L/sw/global/compilers/gcc/5.3.0/lib64 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib64 -L/sw/global/compilers/gcc/5.3.0/lib64 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/libexec/gcc/x86_64-unknown-linux-gnu/5.3.0
-L/sw/global/compilers/gcc/5.3.0/libexec/gcc/x86_64-unknown-linux-gnu/5.3.0 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/sw/global/compilers/gcc/5.3.0/lib/gcc/x86_64-unknown-linux-gnu/5.3.0 -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/x86_64-unknown-linux-gnu/lib
-L/sw/global/compilers/gcc/5.3.0/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/sw/global/compilers/gcc/5.3.0/lib -L/sw/global/compilers/gcc/5.3.0/lib -ldl -Wl,-rpath,/sw/taurus/libraries/openmpi/1.10.2-gnu/lib -lmpi -lgcc_s -lpthread -ldl
<br>
> -----------------------------------------<br>
> <br>
> Regards<br>
> Pietro Incardona<br>
<br>
</div>
</span></font>
</body>
</html>