Matt, thanks for the confirmation.<br><br>Yan<br><br><div class="gmail_quote">On Sat, Sep 19, 2009 at 3:24 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>On Sat, Sep 19, 2009 at 2:12 PM, Ryan Yan <span dir="ltr"><<a href="mailto:vyan2000@gmail.com" target="_blank">vyan2000@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi All,<br>My application code is reading PETSc binary files to obtain the information about a linear system and then solve it in parallel.<br><br>The code works well for median size problem. Now, I am testing a largest case requested by our custom on *one* processor. I got the following errors.<br>
<br>It looks like that error happenned when PETSc is requesting an malloc of size "[0]PETSC ERROR: Memory requested 44784088!", but I did see there are PETSc routines the use even more memory than "44784088",<br>
for instance, "[0] 46 5520000 ISGetIndices_Stride()". So can I guess error is caused by the hardware memory limitation?<br></blockquote></div><div><br>You are running out of memory. If you want to run bigger problems, you will have to use more nodes.<br>
<br> Matt<br> </div><div><div></div><div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">The code was running on MPIS machine with 6 CPUs one each Node.<br>
The code broke for 1 Node with 1 process <br>
for 1 Node with 2 process <br> for 1 Node with 6 process<br><br>But the code succeed for 2 Node with 2 process.<br> for 2 Node with 4 process.<br>
The code also succeed when Node number is big than 2.<br><br>Is this another indicator of the hardware limitation?<br><br>Thanks a lot,<br><br>Yan <br><br><br>$ srun -p sci-comp -N 1 -n 1 ./rpisolve_25_field -ksp_monitor_true_residual -log_summary -malloc_dump -malloc_log >& out.rpisolve.N1.n1 <br>
$ cat out.rpisolve.N1.n1<br><br><br><br><br><br>breakpoint 1<br>breakpoint 2<br>breakpoint 750000<br>[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>[0]PETSC ERROR: Out of memory. This could be due to allocating<br>
[0]PETSC ERROR: too large an object or bleeding by not properly<br>[0]PETSC ERROR: destroying unneeded objects.<br>[0] Maximum memory PetscMalloc()ed 3172769832 maximum size of entire process 0<br>[0] Memory usage sorted by function<br>
[0] 2 3216 ClassPerfLogCreate()<br>[0] 2 1616 ClassRegLogCreate()<br>[0] 2 6416 EventPerfLogCreate()<br>[0] 1 12800 EventPerfLogEnsureSize()<br>[0] 2 1616 EventRegLogCreate()<br>[0] 1 3200 EventRegLogRegister()<br>[0] 92 11960 ISCreateBlock()<br>
[0] 292 36792 ISCreateStride()<br>[0] 46 5520000 ISGetIndices_Stride()<br>[0] 78 21632 KSPCreate()<br>[0] 1 200 KSPCreate_FGMRES()<br>[0] 26 416 KSPDefaultConvergedCreate()<br>[0] 6 17600 KSPSetUp_FGMRES()<br>[0] 475 180880 MatCreate()<br>
[0] 24 3648 MatCreate_MPIAIJ()<br>[0] 71 22152 MatCreate_SeqAIJ()<br>[0] 1 1504 MatGetRow_MPIAIJ()<br>[0] 23 368 MatGetSubMatrices_MPIAIJ()<br>[0] 690 140770488 MatGetSubMatrices_MPIAIJ_Local()<br>[0] 22 5280176 MatGetSubMatrix_MPIAIJ()<br>
[0] 7 1497800024 MatLoad_MPIAIJ()<br>[0] 68 13920000 MatMarkDiagonal_SeqAIJ()<br>[0] 138 1236969200 MatSeqAIJSetPreallocation_SeqAIJ()<br>[0] 23 184 MatSetUpMultiply_MPIAIJ()<br>[0] 24 192 MatStashCreate_Private()<br>[0] 138 1288 MatStashScatterBegin_Private()<br>
[0] 23 184 Mat_CheckCompressedRow()<br>[0] 45 8280360 Mat_CheckInode()<br>[0] 78 14768 PCCreate()<br>[0] 1 120 PCCreate_FieldSplit()<br>[0] 2 208 PCFieldSplitSetDefaults()<br>[0] 50 2400 PCFieldSplitSetFields_FieldSplit()<br>
[0] 1 104 PCSetFromOptions_FieldSplit()<br>[0] 1 200 PCSetUp_FieldSplit()<br>[0] 3 24 PetscCommDuplicate()<br>[0] 1768 84864 PetscFListAdd()<br>[0] 46 368 PetscGatherNumberOfMessages()<br>[0] 237 1896 PetscMapSetUp()<br>
[0] 4 32 PetscMaxSum()<br>
[0] 22 5984 PetscOListAdd()<br>[0] 75 4800 PetscOptionsCreate_Private()<br>[0] 4 96 PetscOptionsGetEList()<br>[0] 6 384000 PetscOptionsInsertFile()<br>[0] 75 600 PetscOptionsInt()<br>[0] 92 736 PetscPostIrecvInt()<br>[0] 46 368 PetscPostIrecvScalar()<br>
[0] 0 32 PetscPushSignalHandler()<br>[0] 4570 130832 PetscStrallocpy()<br>[0] 69 16924048 PetscTableCreate()<br>[0] 1 16 PetscViewerASCIIMonitorCreate()<br>[0] 1 16 PetscViewerASCIIOpen()<br>[0] 12 1952 PetscViewerCreate()<br>
[0] 1 56 PetscViewerCreate_ASCII()<br>[0] 3 192 PetscViewerCreate_Binary()<br>[0] 2 528 StackCreate()<br>[0] 2 1008 StageLogCreate()<br>[0] 2 16 VecAssemblyBegin_MPI()<br>[0] 236 74104 VecCreate()<br>[0] 49 78003168 VecCreate_MPI_Private()<br>
[0] 23 552 VecCreate_Seq_Private()<br>[0] 2 80 VecDuplicateVecs_Default()<br>[0] 92 11224 VecScatterCreate()<br>[0] 72 576 VecStashCreate_Private()<br>[0] 28 1056 VecStashScatterBegin_Private()<br>[0]PETSC ERROR: Memory requested 44784088!<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 5, Mon Apr 13 09:15:37 CDT 2009<br>[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[0]PETSC ERROR: See docs/index.html for manual pages.<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: /tmp/lustre/home/yy2250/local/PETSc/petsc-3.0.0-p5/src/ksp/ksp/examples/tutorials/ttt_5fld/./rpisolve_25_field on a O-hypre-n named sci-m0n0.scsystem by yy2250 Sat Sep 19 14:37:43 2009<br>
[0]PETSC ERROR: Libraries linked from /home/yy2250/local/PETSc/petsc-test-3-p5/O-hypre-nodebug/lib<br>[0]PETSC ERROR: Configure run at Tue Jul 21 15:19:41 2009<br>[0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif77 --with-mpiexec=srun --with-debugging=0 --with-fortran-kernels=generic --with-shared=0 <br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: PetscMallocAlign() line 61 in src/sys/memory/mal.c<br>[0]PETSC ERROR: PetscTrMallocDefault() line 194 in src/sys/memory/mtr.c<br>
[0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 2986 in src/mat/impls/aij/seq/aij.c<br>[0]PETSC ERROR: MatSeqAIJSetPreallocation() line 2928 in src/mat/impls/aij/seq/aij.c<br>[0]PETSC ERROR: MatGetSubMatrices_MPIAIJ_Local() line 1267 in src/mat/impls/aij/mpi/mpiov.c<br>
[0]PETSC ERROR: MatGetSubMatrices_MPIAIJ() line 787 in src/mat/impls/aij/mpi/mpiov.c<br>[0]PETSC ERROR: MatGetSubMatrices() line 5524 in src/mat/interface/matrix.c<br>[0]PETSC ERROR: MatGetSubMatrix_MPIAIJ() line 3069 in src/mat/impls/aij/mpi/mpiaij.c<br>
[0]PETSC ERROR: MatGetSubMatrix() line 6212 in src/mat/interface/matrix.c<br>[0]PETSC ERROR: PCSetUp_FieldSplit() line 285 in src/ksp/pc/impls/fieldsplit/fieldsplit.c<br>[0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>[0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br>[0]PETSC ERROR: main() line 246 in src/ksp/ksp/examples/tutorials/rpisolve_25_field.c<br>
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br>In: PMI_Abort(1, application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0)<br>srun: error: task 0: Exited with exit code 1<br><br>
</blockquote></div></div></div><font color="#888888"><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></blockquote></div><br>