<div dir="ltr">Thanks, Jed. I figured out. I just simply made P more sparse, then the product would not take too much memory.<div><br></div><div>Fande,</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 28, 2018 at 9:59 PM Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It depends entirely on your matrices.  For example, if A is an arrowhead<br>
matrix (graph of a star -- one hub and many leaves) then A^2 is dense.<br>
If you have particular stencils for A and P, then we could tell you the<br>
fill ratio.<br>
<br>
Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> writes:<br>
<br>
> Hi All,<br>
><br>
> I was wondering how much memory is required to get PtAP done? Do you have<br>
> any simple formula to this? So that I can have an  estimate.<br>
><br>
><br>
> Fande,<br>
><br>
><br>
> [132]PETSC ERROR: --------------------- Error Message<br>
> --------------------------------------------------------------<br>
> [132]PETSC ERROR: Out of memory. This could be due to allocating<br>
> [132]PETSC ERROR: too large an object or bleeding by not properly<br>
> [132]PETSC ERROR: destroying unneeded objects.<br>
> [132]PETSC ERROR: Memory allocated 0 Memory used by process 3249920<br>
> [132]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<br>
> [132]PETSC ERROR: Memory requested 89148704<br>
> [132]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> for trouble shooting.<br>
> [132]PETSC ERROR: Petsc Release Version 3.9.4, unknown<br>
> [132]PETSC ERROR: ../../rattlesnake-opt on a arch-theta-avx512-64-opt named<br>
> nid03830 by fdkong Fri Sep 28 22:43:45 2018<br>
> [132]PETSC ERROR: Configure options --LIBS=-lstdc++<br>
> --known-64-bit-blas-indices=0 --known-bits-per-byte=8<br>
> --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8<br>
> --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768<br>
> --known-memcmp-ok=1 --known-mklspblas-supports-zero-based=0<br>
> --known-mpi-c-double-complex=1 --known-mpi-int64_t=1<br>
> --known-mpi-long-double=1 --known-mpi-shared-libraries=0<br>
> --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4<br>
> --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8<br>
> --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8<br>
> --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8<br>
> --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --with-batch=1<br>
> --with-blaslapack-lib="-mkl<br>
> -L/opt/intel/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64"<br>
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC<br>
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn<br>
> --with-fortranlib-autodetect=0 --with-hdf5=0 --with-memalign=64<br>
> --with-mpiexec=aprun --with-shared-libraries=0 --download-metis=1<br>
> --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1<br>
> --download-ptscotch=1 COPTFLAGS="-O3 -xMIC-AVX512" CXXOPTFLAGS="-O3<br>
> -xMIC-AVX512" FOPTFLAGS="-O3 -xMIC-AVX512"<br>
> PETSC_ARCH=arch-theta-avx512-64-opt --with-64-bit-indices=1<br>
> [132]PETSC ERROR: #1 PetscSegBufferCreate() line 64 in<br>
> /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c<br>
> [132]PETSC ERROR: #2 PetscSegBufferCreate() line 64 in<br>
> /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c<br>
> [132]PETSC ERROR: #3 PetscSegBufferExtractInPlace() line 227 in<br>
> /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c<br>
> [132]PETSC ERROR: #4 MatStashScatterBegin_BTS() line 854 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c<br>
> [132]PETSC ERROR: #5 MatStashScatterBegin_Private() line 461 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c<br>
> [132]PETSC ERROR: #6 MatAssemblyBegin_MPIAIJ() line 683 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiaij.c<br>
> [132]PETSC ERROR: #7 MatAssemblyBegin() line 5158 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c<br>
> [132]PETSC ERROR: #8 MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() line 262 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c<br>
> [132]PETSC ERROR: #9 MatPtAP_MPIAIJ_MPIAIJ() line 172 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c<br>
> [132]PETSC ERROR: #10 MatPtAP() line 9182 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c<br>
> [132]PETSC ERROR: #11 MatGalerkin() line 10615 in<br>
> /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c<br>
> [132]PETSC ERROR: #12 PCSetUp_MG() line 730 in<br>
> /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/mg/mg.c<br>
> [132]PETSC ERROR: #13 PCSetUp_HMG() line 336 in<br>
> /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/hmg/hmg.c<br>
> [132]PETSC ERROR: #14 PCSetUp() line 923 in<br>
> /gpfs/mira-home/fdkong/petsc/src/ksp/pc/interface/precon.c<br>
> [132]PETSC ERROR: #15 KSPSetUp() line 381 in<br>
> /gpfs/mira-home/fdkong/petsc/src/ksp/ksp/interface/itfunc.c<br>
> [136]PETSC ERROR: --------------------- Error Message<br>
> ----------------------------------------------------------<br>
</blockquote></div>