<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, May 1, 2015 at 4:21 AM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Dear All,<br>
<br>
I have run my codes successfully with up to 100 million total
unknowns using 1000 processors on WestGrid Jasper Cluster, Canada.
But when I scale the unknows up to 1 billion, the codes crashes with
the following error. It's out of memory.<br></div></blockquote><div><br></div><div>If you are running out of memory, you need to use more processors.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<font color="#ff0000">Error message from valgrind output</font><br>
<br>
==9344== Invalid read of size 16<br>
==9344== at 0xADB2906: __intel_sse2_strdup (in
/lustre/jasper/software/intel/l_ics_2013.0.028/composer\<br>
_xe_2013.1.117/compiler/lib/intel64/libintlc.so.5)<br>
==9344== by 0xE6: ???<br>
==9344== by 0xE7: ???<br>
==9344== by 0x5: ???<br>
==9344== Address 0xb364410 is 16 bytes inside a block of size 17
alloc'd<br>
==9344== at 0x4A0638D: malloc (vg_replace_malloc.c:291)<br>
==9344== by 0x3DE7C6807C: vasprintf (in /lib64/<a href="http://libc-2.5.so" target="_blank">libc-2.5.so</a>)<br>
==9344== by 0x3DE7C4CBE7: asprintf (in /lib64/<a href="http://libc-2.5.so" target="_blank">libc-2.5.so</a>)<br>
==9344== by 0x9DC511E: opal_output_init (output.c:144)<br>
==9344== by 0x9DC042D: opal_init_util (opal_init.c:207)<br>
==9344== by 0x9CF4EBB: ompi_mpi_init (ompi_mpi_init.c:309)<br>
==9344== by 0x9D0D802: PMPI_Init (pinit.c:84)<br>
==9344== by 0x905E976: PMPI_INIT (pinit_f.c:75)<br>
==9344== by 0x4D5280F: petscinitialize_ (in
/lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.\<br>
3.5.1)<br>
==9344== by 0x439D05: petsc_mpi_common_mp_petsc_mpi_initialize_
(in /lustre/home/danyangs/benchmark/ba\<br>
sin/min3p_thcm)<br>
==9344== by 0x5FDBB9: MAIN__ (in
/lustre/home/danyangs/benchmark/basin/min3p_thcm)<br>
==9344== by 0x4321FB: main (in
/lustre/home/danyangs/benchmark/basin/min3p_thcm)<br>
==9344==<br>
<br>
<font color="#ff0000">Error message from Jasper Cluster output</font><br>
--32725:0:aspacem <<< SHOW_SEGMENTS: out_of_memory (407
segments, 96 segnames)<br>
--32725:0:aspacem ( 0)
/lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/memcheck-amd64-linux<br>
--32725:0:aspacem ( 1)
/lustre/home/danyangs/benchmark/basin/min3p_thcm<br>
--32725:0:aspacem ( 2) /lib64/<a href="http://ld-2.5.so" target="_blank">ld-2.5.so</a><br>
--32725:0:aspacem ( 3)
/data2/PBStmp/<a href="http://6456165.jasper-usradm.westgrid.ca/vgdb-pipe-shared-mem-vgdb-32725-b%5C" target="_blank">6456165.jasper-usradm.westgrid.ca/vgdb-pipe-shared-mem-vgdb-32725-b\</a><br>
y-danyangs-on-cl2n050<br>
--32725:0:aspacem ( 4)
/lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_core-amd64\<br>
-linux.so<br>
--32725:0:aspacem ( 5)
/lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_memcheck-a\<br>
md64-linux.so<br>
--32725:0:aspacem ( 6)
/lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.3.5.1<br>
--32725:0:aspacem ( 7)
/lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_cxx.so.1.0.2<br>
--32725:0:aspacem ( 8)
/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\<br>
el64/libmkl_scalapack_lp64.so<br>
--32725:0:aspacem ( 9)
/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\<br>
el64/libmkl_intel_lp64.so<br>
--32725:0:aspacem (10)
/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\<br>
el64/libmkl_sequential.so<br>
--32725:0:aspacem (11)
/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\<br>
el64/libmkl_core.so<br>
--32725:0:aspacem (12)
/lustre/jasper/software/petsc/petsc-3.5.1/lib/libparmetis.so<br>
--32725:0:aspacem (13)
/lustre/jasper/software/petsc/petsc-3.5.1/lib/libmetis.so<br>
--32725:0:aspacem (14)
/lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/openmpi/mca_paffinity_hwl\<br>
oc.so<br>
--32725:0:aspacem (15) /usr/lib64/libX11.so.6.2.0<br>
--32725:0:aspacem (16) /lib64/<a href="http://libpthread-2.5.so" target="_blank">libpthread-2.5.so</a><br>
--32725:0:aspacem (17) /lib64/libssl.so.0.9.8e<br>
--32725:0:aspacem (18) /lib64/libcrypto.so.0.9.8e<br>
--32725:0:aspacem (19)
/lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f90.so.1.3.0<br>
--32725:0:aspacem (20)
/lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f77.so.1.0.7<br>
--32725:0:aspacem (21)
/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/compiler/li\<br>
b/intel64/libimf.so<br>
<br>
<font color="#ff0000">The PETSc configuration is as below</font><br>
<br>
================================================================================<br>
Starting Configure Run at Tue Dec 16 10:42:20 2014<br>
Configure Options: --configModules=PETSc.Configure
--optionsModule=PETSc.compilerOptions --prefix=/global\<br>
/software/petsc/petsc-3.5.1 --with-shared-libraries
--with-mpirun=mpiexec --with-vendor-compiler=intel --\<br>
with-blas-lapack-lib=-mkl=sequential --with-cc=mpicc
--with-cxx=mpiCC --with-fc=mpif90 --COPTFLAGS=-O2 --\<br>
CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-debugging=no --with-blacs=yes
--with-blacs-include=/lustre/jasper/\<br>
software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/include
--with-blacs-lib=/lustre/jasper/softwa\<br>
re/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a
--with-scala\<br>
pack=yes
--with-scalapack-include=/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/m\<br>
kl/include
--with-scalapack-lib="-L/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/\<br>
mkl/lib/intel64/ -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"
--download-metis=metis-5.0.2-p3.tar.gz --\<br>
download-parmetis=yes --download-superlu_dist=yes
--download-hypre=yes<br>
Working directory: /lustre/jasper/software-build/petsc/petsc-3.5.1<br>
Machine platform:<br>
('Linux', '<a href="http://jasper.westgrid.ca" target="_blank">jasper.westgrid.ca</a>', '2.6.18-274.el5', '#1 SMP Fri Jul 22
04:43:29 EDT 2011', 'x86_64', 'x86_6\<br>
4')<br>
Python version:<br>
2.4.3 (#1, Sep 21 2011, 19:55:41)<br>
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)]<br>
================================================================================<br>
<br>
My case does not hold the conditions of 64-bit-indices as below.<br>
<span style="color:rgb(0,0,0);font-family:Monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(213,234,255)">By default the type
that PETSc uses to index into arrays and keep sizes of arrays is a
PetscInt defined to be a 32 bit int. If your problem</span>
<ul style="color:rgb(0,0,0);font-family:Monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(213,234,255)">
<li>involves more than 2^31 - 1 unknowns (around 2 billion) OR</li>
<li>your matrix might contain more than 2^31 - 1 nonzeros on a
single process</li>
</ul>
<span style="color:rgb(0,0,0);font-family:Monospace;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(213,234,255)">then you need to use
this option. Otherwise you will get strange crashes.</span><br>
<br>
Do you guys have suggestions on this?<br>
<br>
Thanks and regards,<br>
<br>
Danyang<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>