[petsc-users] MPI linear solver reproducibility question

[Jed Brown]

Mark McClure <mark at resfrac.com> writes:

> Thank you, I will try BCGSL.
>
> And good to know that this is worth pursuing, and that it is possible. Step
> 1, I guess I should upgrade to the latest release on Petsc.
>
> How can I make sure that I am "using an MPI that follows the suggestion for
> implementers about determinism"? I am using MPICH version 3.3a2.
>
> I am pretty sure that I'm assembling the same matrix every time, but I'm
> not sure how it would depend on 'how you do the communication'. Each
> process is doing a series of MatSetValues with INSERT_VALUES,
> assembling the matrix by rows. My understanding of this process is that
> it'd be deterministic.

In the typical FD implementation, you only set local rows, but with FE and sometimes FV, you also create values that need to be communicated and summed on other processors.