[petsc-users] Troubles updating my code from PETSc-3.4 to 3.5 Using MUMPS for KSPSolve()

Fri Dec 19 11:25:30 CST 2014

   Marc,

    The "correct", but time consuming way, to determine exactly what code change changed your result is to use git bisection with the two endpoints of maint-3.4 and maint. You'll need to read up on running it but essentially you configure and run with maint-3.4 confirm your one result, then configure and run with maint and confirm the result is something else. Then git will bisect through all the changes giving you new places to configure and run until it finds the exact change set that changed the results. But this will be time consuming because to really check you should rerun configure and make all each time. It will eventually find the change that changed your solution.

   Before doing that I checked a few things. Both maint-3.4 and maint use the same versions of the external packages:

'http://ftp.mcs.anl.gov/pub/petsc/externalpackages/MUMPS_4.10.0-p3.tar.gz']
'http://ftp.mcs.anl.gov/pub/petsc/externalpackages/parmetis-4.0.2-p5.tar.gz'
'https://gforge.inria.fr/frs/download.php/31832/scotch_6.0.0_esmumps.tar.gz',
http://www.netlib.org/scalapack/scalapack-2.0.2.tgz',

  I also checked out out the PETSc interface to mumps: called mumps.c for both maint and maint-3.4 and did a diff (all attached) it is mostly just adding options but perhaps it changes something. 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mumps.c
Type: application/octet-stream
Size: 75025 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20141219/279caf37/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mumps.c-3.4
Type: application/octet-stream
Size: 64792 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20141219/279caf37/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mumps.c-diff
Type: application/octet-stream
Size: 16670 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20141219/279caf37/attachment-0005.obj>
-------------- next part --------------
I suggest starting with maint-3.4 and changing the one mumps.c file to the new one (mumps.c in the attachment) and see if your application gives the old or new result. This will tell if something changed in mumps.c to make the solution change. We did not consciously change anything (that I can remember or am aware of) that would change the parameters MUMPS is called with but ....

   Good luck and let us know how it goes,

   Barry

> On Dec 19, 2014, at 5:39 AM, Marc MEDALE <marc.medale at univ-amu.fr> wrote:
> 
> 
> After any changes or upgrade to new PETSc versions I use to run tests which compare the outputs between previous release and the new one. And I was very surprised to see such huge differences in outputs, for components that haven't been told to be changed in the changes_3.5 log file (neither KSP nor MUMPS were told to be changed in the changes from PETSc-3.4 to 3.5).
> 
> So, as longtime PETSc user (and aficionado), I would greatly appreciate you give me some hints about the following fundamental question: what are the changes that have been done in PETSc to change in such a way the solution computed by the MUMPS solver throughout the PETSc interface?
> 
> Thank you again for your time and pertinent explanations, as usual. Have a good day.
> 
> Marc MEDALE
> 
> 
> 
> 
> 
> Le 19 déc. 2014 à 00:29, Barry Smith <bsmith at mcs.anl.gov> a écrit :
> 
>> 
>>   Marc,
>> 
>>    I have played around with your matrix using several "direct" solvers; all of them produced residual norms of around 10^-8 except Matlab which produced a residual norm of 10^-5 and a warning that the recond of the matrix was 10^-16
>> I also solved with the PETSc LU factorization in quad precision and got a residual norm of 10^-26. UMFPACK ran out of memory.
>> 
>>    Attached I plotted the solution (as a 1d vector for all the solvers), as you can see all the answers are very different. Note that the figures are black inside the "envelop" of the solution because intermediate values of the vector are dense (ie. at a very fine scale the solution is oscillating a great deal). Interesting when I ran with MUMPS on 1 and 2 processors using PETSc 3.5p2 (actually the maint branch) I get a solution very near your "old" solution. 
>> 
>>   IThe matrix has 8,000 rows of the identity (1 on the diagonal) and the rest with entries of -10^8.  Don't put those trivial rows into the matrix and scale the matrix so it has positive diagonal entires. I've done this and it doesn't help the solver but it is still the right thing to do.
>> 
>> <Untitled.png>
>> 
>>    I do not know if the quad precision solution is "accurate" but I am pretty confident that all the other answers are equally valid and equally worthless. How do you know the "old" mumps solution is correct and the new mumps solution wrong?
>> 
>>   Can you try with the maint branch of PETSc and see if you get the solutions you want?
>> 
>> 
>>   Barry
>> 
>> 
>> 
>>> On Dec 11, 2014, at 4:38 AM, Marc MEDALE <marc.medale at univ-amu.fr> wrote:
>>> 
>>> Dear PETSC Users,
>>> 
>>> I have just updated to PETSc-3.5 my research code that uses PETSc for a while but I'm facing an astonishing difference between PETSc-3.4 to 3.5 versions when solving a very ill conditioned algebraic system with MUMPS (4.10.0 in both cases).
>>> 
>>> The only differences that arise in my fortran source code are the following:
>>> Loma1-medale% diff ../version_3.5/solvEFL_MAN_SBIF.F ../version_3.4/solvEFL_MAN_SBIF.F
>>> 336,337d335
>>> <       CALL MatSetOption(MATGLOB,MAT_KEEP_NONZERO_PATTERN,
>>> <      &       PETSC_TRUE,IER)
>>> 749,750c747,748
>>> <       CALL KSPSetTolerances(KSP1,TOL,PETSC_DEFAULT_REAL,
>>> <      &     PETSC_DEFAULT_REAL,PETSC_DEFAULT_INTEGER,IER)
>>> ---
>>>>      CALL KSPSetTolerances(KSP1,TOL,PETSC_DEFAULT_DOUBLE_PRECISION,
>>>>     &     PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,IER)
>>> 909c907,908
>>> <           CALL KSPSetOperators(KSP1,MATGLOB,MATGLOB,IER)
>>> ---
>>>>          CALL KSPSetOperators(KSP1,MATGLOB,MATGLOB,
>>>>     &                         SAME_NONZERO_PATTERN,IER)
>>> 
>>> When I run the corresponding program versions on 128 cores of our cluster with the same input data and the following command line arguments:
>>> -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_8 0
>>> 
>>> I get the following outputs:
>>> a)  with PETSc-3.4p4:
>>> 	L2 norm of solution vector: 7.39640E-02,
>>> 
>>> b)  with PETSc-3.5p1:
>>> 	L2 norm of solution vector: 1.61325E-02
>>> 
>>> Do I have change something else in updating my code based on KSP from PETSc-3.4 to 3.5 versions?
>>> Do any default values in the PETSc-MUMPS interface have been changed from PETSc-3.4 to 3.5?
>>> Any hints or suggestions are welcome to help me to recover the right results (obtained with PETSc-3.4).
>>> 
>>> Thank you very much.
>>> 
>>> Marc MEDALE.
>> 
> 