[petsc-users] On the edge of 2^31 unknowns

[Eric Chamberland]

Hi,

we just tried this morning to solve a 1.7 billion dofs problem (on 1060 
processors task)  .

Unfortunately, petsc-3.5.3 didn't succeed within a MatMatMult product 
with the following backtrace:

[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
[0]PETSC ERROR: Memory allocated 0 Memory used by process 4126371840
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[0]PETSC ERROR: Memory requested 18446744070461249536
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015
[0]PETSC ERROR: /rap/jsf-051-aa/ericc/GIREF/bin/probGD.opt on a 
linux-gnu-intel named r101-n31 by
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-intel CFLAGS="-O3 
-xHost -mkl -fPIC -m64 -n
[0]PETSC ERROR: #1 PetscMallocAlign() line 46 in 
/software6/src/petsc-3.5.3/src/sys/memory/mal.c
[0]PETSC ERROR: #2 PetscLLCondensedCreate_Scalable() line 1327 in 
/software6/src/petsc-3.5.3/inclu
[0]PETSC ERROR: #3 MatMatMultSymbolic_MPIAIJ_MPIAIJ() line 741 in 
/software6/src/petsc-3.5.3/src/m
[0]PETSC ERROR: #4 MatMatMult_MPIAIJ_MPIAIJ() line 33 in 
/software6/src/petsc-3.5.3/src/mat/impls/
[0]PETSC ERROR: #5 MatMatMult() line 8713 in 
/software6/src/petsc-3.5.3/src/mat/interface/matrix.c

The amount of memory requested is, apparently, an overflow of a PetscInt 
which is a 32 bit signed int.

Bad user I am, I didn't expected that to solve a problem with less than 
2^31 unknowns, it could be mandatory to compile petsc with 64bit indices...

Is compiling with "--with-64-bit-indices" the only solution to my problem?

Is it a known limitation/bug with a "Petsc-32bit-indices"?

Thanks,

Eric