[petsc-users] Errors from large matrices

[Barry Smith]

On May 26, 2013, at 1:10 PM, Choi240 <choi240 at purdue.edu> wrote:

> Thank you for your fast reply. However, I was already using 64 bit pointer.

    64 bit pointers are very very different than --with-64-bit-indices!  With 64 bit pointers integers are still 32 bits which means one cannot have an integer number greater than about 2 billion. So if your matrix dimension is more than about 2 billion or you have over 2 billion non zeros (sequential) then you need to configure PETSc with --with-64-bit-indices.

  Based on your statement size:4273949x108965941330383 you need to configure with --with-64-bit-indices. Now did you configure with 64 bit indices? 

   Barry

> 
> 
> -------- Original message --------
> Subject: Re: [petsc-users] Errors from large matrices 
> From: Barry Smith <bsmith at mcs.anl.gov> 
> To: Joon hee Choi <choi240 at purdue.edu> 
> CC: petsc-users at mcs.anl.gov 
> 
> 
> 
> On May 26, 2013, at 6:19 AM, Joon hee Choi <choi240 at purdue.edu> wrote:
> 
> > Hello all,
> > 
> > I need to multiply a large seqaij matrix(X1) and a maij(or baij) matrix(CC). I set up X1 (size:4273949x108965941330383, nonzeros:143599552) and C (size:25495389x10, nonzeros:254953890) and created a maij matrix CC from C. However, I got errors such as out of memory and Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range. Is this memory problem, and do I have to change seqaij into mpiaij and use multi processors? Or do I have another methods fixing it? If you know the method, then please let me know it. Thank you.
> 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#with-64-bit-indices
> 
> 
> > 
> > Joon
> > 
> > 
> > Code:
> >  ...
> >  ierr = MatCreate(PETSC_COMM_SELF, &X1); CHKERRQ(ierr);
> >  ierr = MatSetSizes(X1, PETSC_DECIDE, PETSC_DECIDE, I, J*K); CHKERRQ(ierr);
> >  ierr = MatSetBlockSizes(X1, I, J); CHKERRQ(ierr);
> >  ierr = MatSetType(X1, MATSEQAIJ); CHKERRQ(ierr);
> >  ierr = MatSeqAIJSetPreallocation(X1, 0, nnz); CHKERRQ(ierr);
> > 
> >  for (int x=0; x<tups.size(); x++) {
> >       i = std::tr1::get<0>(tups[x]);
> >       j = std::tr1::get<2>(tups[x]) + std::tr1::get<1>(tups[x])*J;
> >       val = std::tr1::get<3>(tups[x]);
> >       ierr = MatSetValues(X1, 1, &i, 1, &j, &val, INSERT_VALUES); CHKERRQ(ierr);
> >  }
> >  ierr = MatAssemblyBegin(X1, MAT_FINAL_ASSEMBLY);
> >  ierr = MatAssemblyEnd(X1, MAT_FINAL_ASSEMBLY);
> >  ierr = PetscGetTime(&v1); CHKERRQ(ierr);
> >  ierr = PetscPrintf(PETSC_COMM_WORLD, "Setup Time: %2.1e \n", v1-v); CHKERRQ(ierr);
> > 
> >  // Create a matrix C (K x R) with all values 1
> >  ierr = MatCreateSeqAIJ(PETSC_COMM_SELF, K, R, R, NULL, &C); CHKERRQ(ierr);
> >  for (k=0; k<K; k++) {
> >       for (r=0; r<R; r++) {
> >            ierr = MatSetValues(C, 1, &k, 1, &r, &one, INSERT_VALUES); CHKERRQ(ierr);
> >       }
> >  }
> >  ierr = MatAssemblyBegin(C, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> >  ierr = MatAssemblyEnd(C, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> > 
> >  ierr = M...