[petsc-users] segfault in MatAssemblyEnd() when using large matrices on multi-core MAC OS-X

[Jed Brown]

I ran

$ mpirun -n 2 ./a.out
 ---------------------------------------------------
 Running MAS PETSc tests with:
 nr:                   23
 np:                   27
 nt:                   25
 N:                 46575
 num_steps:          2000
 MPI cores:             2
 ---------------------------------------------------
 MPI TEST:  My rank is:           0
 MPI TEST:  My rank is:           1
 Rank            0  has rows            0  to        23288
 Rank            1  has rows        23288  to        46575
 Number of non-zero entries in matrix:       690339
 Computing y=Ax with RON_CSR_AX...
 ...done!
 ||y||:     821.67460825997637
 y(5)=      2.7454534359667053
 Storing MAS matrix into PETSc matrix...
 ...done!
 rank            1  about to call MatAssemblyEnd()...
 rank            0  about to call MatAssemblyEnd()...
 Computing y=Ax with PETSc MatMult...
 ...done!
 ||y||:     821.67460825997568
 y(5)=      2.7454534359667053
 RON_CSR_AX y=Ax time:      201.704498     nsec/mp.
 PETSc y=Ax time:           198.269424     nsec/mp.
 PETSc y=Ax flops:         0.417739183     GFLOPS.
 Did not converge! Number of iteratioins:            0


On Mon, Jul 30, 2012 at 3:12 PM, Ronald M. Caplan <caplanr at predsci.com>wrote:

> Attached is the code.  The original code which segfaults with more than
> one core is the code I sent last week.
>
>  - Ron C
>
>
> On Mon, Jul 30, 2012 at 3:09 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Mon, Jul 30, 2012 at 3:04 PM, Ronald M. Caplan <caplanr at predsci.com>wrote:
>>
>>> I seem to have solved the problem.
>>>
>>> I was storing my entire matrix on node 0 and then calling MatAssembly
>>> (begin and end) on all nodes (which should have worked...).
>>>
>>> Apparently I was using too much space for the buffering or the like,
>>> because when I change the code so each node sets its own matrix values,
>>> than the MatAssemblyEnd does not seg fault.
>>>
>>
>> Can you send the test case. It shouldn't seg-fault unless the machine
>> runs out of memory (and most desktop systems have overcommit, so the system
>> will kill arbitrary processes, not necessarily the job that did the latest
>> malloc.
>>
>> In practice, you should call MatAssemblyBegin(...,MAT_FLUSH_ASSEMBLY)
>> periodically.
>>
>>
>>>
>>> Why should this be the case?   How many elements of a vector or matrix
>>> can a single node "set" before Assembly to distribute over all nodes?
>>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120730/02f3ed55/attachment.html>