[petsc-users] Question on MatCreateSeqAIJ() and 32-bit integer

[Karl Rupp]

Hi,

yes, you really need to re-compile the PETSc library to get 64-bit 
integers. You can build the library below your home folder, there is no 
need to beg your admin for this. :-)

(As an additional benefit, you can also enable additional packages which 
may not be available in your default installation).

Best regards,
Karli


On 07/30/2013 02:05 PM, Lu Qiyue wrote:
> Hello Barry:
> Switching to 64 bit, means to re-install PETSC and add
> --with-64-bit-indices
> when do the configuration, right?
>
> Because re-install PETSC is not easy here and I am wondering, is there
> any other way to solve this problem? Like add some ' -bigint' option
> while compiling the driver? Looks not enough for this case, because
> PETSC still at 32-bits.
>
>  From what you said that 'because of internal counts inside of PETSc
> (actually the way we store the matrices) it requires 64 bit indices', I
> am afraid that re-install is the only solution?
>
> Thanks
>
> Qiyue Lu
>
>
>
> On Tue, Jul 30, 2013 at 1:38 PM, Barry Smith <bsmith at mcs.anl.gov
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>        Lu,
>
>          You need to switch to 64 bit indices. Even though the
>     individual number of non zeros per row is less than the 32 bit limit
>     the total number of non zeros in the matrix is higher than the
>     limit; how much physical memory you have is not relevant. As you
>     note it is because of internal counts inside of PETSc (actually the
>     way we store the matrices) it requires 64 bit indices.
>
>         Barry
>
>     Switching to 64 bit indices should not be difficult, please let us
>     know if you have any problems.
>
>     On Jul 30, 2013, at 10:54 AM, Lu Qiyue <luqiyue at gmail.com
>     <mailto:luqiyue at gmail.com>> wrote:
>
>      > Dear All:
>      >
>      > I am solving a huge system AX=b, A is a sparse matrix but its
>     number of non-zeros is 3.29*10^9. I noticed that the 32-bit integer
>     upper limit is ~2.15*10^9.
>      >
>      > A is in COO format here. When I prepare the input *.bin file for
>     Petsc, the line
>      >
>      > ierr = MatCreateSeqAIJ(PETSC_COMM_WORLD,m,n,0,cnt,&A);CHKERRQ(ierr);
>      >
>      > can not pass.
>      >
>      > m =n =39979380 here, cnt is an array holding the number of
>     non-zeros per row. The error message is:
>      >
>      > [0]PETSC ERROR: Out of memory. This could be due to allocating
>      > [0]PETSC ERROR: too large an object or bleeding by not properly
>      > [0]PETSC ERROR: destroying unneeded objects.
>      > [0]PETSC ERROR: Memory allocated 0 Memory used by process 492802048
>      > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for
>     info.
>      > [0]PETSC ERROR: Memory requested 18446744061813993472!
>      >
>      > Calling MatCreateSeqAIJ() doesn't involve NNZ information there,
>     m/n/cnt are all integers less than 32-bit integer limit.
>      > And the total size of data in COO format: nnz*8
>     bytes(values)+nnz*4 bytes(rows)+nnz*4 bytes(cols) are less than the
>     memory limit of our system.
>      >
>      > The code works on a system with half size of this failed one.
>      >
>      > I am wondering, Does this because MatCreateSeqAIJ() might do some
>     'internal' counting things which exceeds the integer limit in this case?
>      >
>      > Thanks
>      >
>      > Qiyue Lu
>
>