From ncreati at inogs.it  Mon Apr  1 08:07:16 2019
From: ncreati at inogs.it (Nicola Creati)
Date: Mon, 1 Apr 2019 15:07:16 +0200
Subject: [petsc-users] petsc4py - Convert c example code to Python
Message-ID: <0e04b6f0-2cb0-e021-7f8c-4271cd7eea42@inogs.it>

Hello,

thanks Barry. I cannot test both codes, C and Python, since the filling 
of the Jacobian matrix? get the following Error:

Traceback (most recent call last):
 ? File "PETSc/petscts.pxi", line 382, in petsc4py.PETSc.TS_IJacobian
 ? File "ex13.py", line 61, in Jacobian_func
 ??? B.setValueStencil(row, col, value)
 ? File "PETSc/Mat.pyx", line 948, in petsc4py.PETSc.Mat.setValueStencil
 ? File "PETSc/petscmat.pxi", line 1100, in 
petsc4py.PETSc.matsetvaluestencil
petsc4py.PETSc.Error: error code 63
[0] TSComputeIJacobian() line 912 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
[0] MatSetValuesStencil() line 1576 in 
/home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
[0] MatSetValuesLocal() line 2152 in 
/home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
[0] MatSetValues() line 1349 in 
/home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
[0] MatSetValues_SeqAIJ() line 477 in 
/home/ncreati/Downloads/petsc-3.10.3/src/mat/impls/aij/seq/aij.c
[0] Argument out of range
[0] Inserting a new nonzero at (9,8) in the matrix

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x7fa47b53c0c8> 
returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x7fa47b53c0c8> 
returned a result with an error set
Traceback (most recent call last):
 ? File "ex13.py", line 121, in <module>
 ??? ts.solve(u)
 ? File "PETSc/TS.pyx", line 657, in petsc4py.PETSc.TS.solve
petsc4py.PETSc.Error: error code 73
[0] TSSolve() line 3775 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
[0] TSStep() line 3592 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
[0] TSStep_Theta() line 216 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ts/impls/implicit/theta/theta.c
[0] TSTheta_SNESSolve() line 176 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ts/impls/implicit/theta/theta.c
[0] SNESSolve() line 4397 in 
/home/ncreati/Downloads/petsc-3.10.3/src/snes/interface/snes.c
[0] SNESSolve_NEWTONLS() line 224 in 
/home/ncreati/Downloads/petsc-3.10.3/src/snes/impls/ls/ls.c
[0] KSPSolve() line 723 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0] KSPSetUp() line 391 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0] PCSetUp() line 932 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ksp/pc/interface/precon.c
[0] PCSetUp_ILU() line 134 in 
/home/ncreati/Downloads/petsc-3.10.3/src/ksp/pc/impls/factor/ilu/ilu.c
[0] MatGetOrdering() line 180 in 
/home/ncreati/Downloads/petsc-3.10.3/src/mat/order/sorder.c
[0] Object is in wrong state
[0] Not for unassembled matrix

I believe that my Python conversion is wrong but I cannot figure it out.

Nicola


-- 

Nicola Creati
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
off.   +39 040 2140 213
fax.   +39 040 327307

_____________________________________________________________________
This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.


From knepley at gmail.com  Mon Apr  1 08:17:38 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Apr 2019 09:17:38 -0400
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <0e04b6f0-2cb0-e021-7f8c-4271cd7eea42@inogs.it>
References: <0e04b6f0-2cb0-e021-7f8c-4271cd7eea42@inogs.it>
Message-ID: <CAMYG4Gn_i077qnkFPO3FmG9hMdcaS-d6MgtsNYoUGowkucLkEw@mail.gmail.com>

On Mon, Apr 1, 2019 at 9:08 AM Nicola Creati via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> thanks Barry. I cannot test both codes, C and Python, since the filling
> of the Jacobian matrix  get the following Error:
>
> Traceback (most recent call last):
>    File "PETSc/petscts.pxi", line 382, in petsc4py.PETSc.TS_IJacobian
>    File "ex13.py", line 61, in Jacobian_func
>      B.setValueStencil(row, col, value)
>    File "PETSc/Mat.pyx", line 948, in petsc4py.PETSc.Mat.setValueStencil
>    File "PETSc/petscmat.pxi", line 1100, in
> petsc4py.PETSc.matsetvaluestencil
> petsc4py.PETSc.Error: error code 63
> [0] TSComputeIJacobian() line 912 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
> [0] MatSetValuesStencil() line 1576 in
> /home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
> [0] MatSetValuesLocal() line 2152 in
> /home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
> [0] MatSetValues() line 1349 in
> /home/ncreati/Downloads/petsc-3.10.3/src/mat/interface/matrix.c
> [0] MatSetValues_SeqAIJ() line 477 in
> /home/ncreati/Downloads/petsc-3.10.3/src/mat/impls/aij/seq/aij.c
> [0] Argument out of range
> [0] Inserting a new nonzero at (9,8) in the matrix
>

This comes from improper preallocation of the matrix. You can call

  J.setOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)

to turn it off I believe. If you created this matrix yourself, then its
probably
just a logic error in preallocation. If you got it from a DM, then you a
trying to insert outside the stencil you told the DM about.

  Thanks,

    Matt


> The above exception was the direct cause of the following exception:
>
> SystemError: <built-in method insert of list object at 0x7fa47b53c0c8>
> returned a result with an error set
> Exception ignored in: 'petsc4py.PETSc.traceback'
> SystemError: <built-in method insert of list object at 0x7fa47b53c0c8>
> returned a result with an error set
> Traceback (most recent call last):
>    File "ex13.py", line 121, in <module>
>      ts.solve(u)
>    File "PETSc/TS.pyx", line 657, in petsc4py.PETSc.TS.solve
> petsc4py.PETSc.Error: error code 73
> [0] TSSolve() line 3775 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
> [0] TSStep() line 3592 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ts/interface/ts.c
> [0] TSStep_Theta() line 216 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ts/impls/implicit/theta/theta.c
> [0] TSTheta_SNESSolve() line 176 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ts/impls/implicit/theta/theta.c
> [0] SNESSolve() line 4397 in
> /home/ncreati/Downloads/petsc-3.10.3/src/snes/interface/snes.c
> [0] SNESSolve_NEWTONLS() line 224 in
> /home/ncreati/Downloads/petsc-3.10.3/src/snes/impls/ls/ls.c
> [0] KSPSolve() line 723 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0] KSPSetUp() line 391 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0] PCSetUp() line 932 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ksp/pc/interface/precon.c
> [0] PCSetUp_ILU() line 134 in
> /home/ncreati/Downloads/petsc-3.10.3/src/ksp/pc/impls/factor/ilu/ilu.c
> [0] MatGetOrdering() line 180 in
> /home/ncreati/Downloads/petsc-3.10.3/src/mat/order/sorder.c
> [0] Object is in wrong state
> [0] Not for unassembled matrix
>
> I believe that my Python conversion is wrong but I cannot figure it out.
>
> Nicola
>
>
> --
>
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS
> www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere
> Department
> CARS (Cartography and Remote Sensing) Research Group
> http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY
> ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
>
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally
> privileged information, is intended solely for the use of the intended
> addressees. Opinions, conclusions and other information contained in this
> message, that do not relate to the official business of OGS, shall be
> considered as not given or endorsed by it. Every opinion or advice
> contained in this communication is subject to the terms and conditions
> provided by the agreement governing the engagement with such a client. Any
> use, disclosure, copying or distribution of the contents of this
> communication by a not-intended recipient or in violation of the purposes
> of this communication is strictly prohibited and may be unlawful. For Italy
> only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le
> informazioni contenute in questo messaggio sono riservate ed a uso
> esclusivo del destinatario.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/6b306fce/attachment.html>

From yjwu16 at gmail.com  Mon Apr  1 09:20:25 2019
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Mon, 1 Apr 2019 22:20:25 +0800
Subject: [petsc-users] Problem about Residual evaluation
Message-ID: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>

Dear PETSc developers:
Hi,

I've been using -snes_mf_operator and I've customized a precondition matrix
to solve my problem.I have two questions about the residuals of linear
steps(KSP residual).


1.Since I'm using a matrix-free method, how do we get KSP residuals in
PETSc?

r_m = b - A*x_m

Is finite difference used to approximate "A*x_m" ?


2.What is the difference between instruction ' -ksp_monitor ' and '
-ksp_monitor_true_residual ' in how they are calculated?


Thanks,

Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/4a060ebb/attachment.html>

From ncreati at inogs.it  Mon Apr  1 09:45:58 2019
From: ncreati at inogs.it (Nicola Creati)
Date: Mon, 1 Apr 2019 16:45:58 +0200
Subject: [petsc-users] petsc4py - Convert c example code to Python
Message-ID: <6e05c686-b3a7-7015-ff27-9772881979ef@inogs.it>

Thanks Matt, it solves the problem. The Jacobian matrix is now filled 
but checking between the C and Python code with MatView there are some 
differences.
This is my Jacobian matrix:

Mat Object: 1 MPI processes
 ? type: seqaij
row 0: (0, 1.)
row 1: (1, 1.)
row 2: (2, 1.)
row 3: (3, 1.)
row 4: (4, 1.)
row 5: (5, 1.)
row 6: (6, 1.)
row 7: (7, 1.)
row 8: (8, 1.)
row 9: (1, 49.)? (8, 49.)? (9, -196.)? (10, 49.)? (17, 49.)
row 10: (2, 49.)? (9, 49.)? (10, -196.)? (11, 49.)? (18, 49.)
row 11: (3, 49.)? (10, 49.)? (11, -196.)? (12, 49.)? (19, 49.)
row 12: (4, 49.)? (11, 49.)? (12, -196.)? (13, 49.)? (20, 49.)
row 13: (5, 49.)? (12, 49.)? (13, -196.)? (14, 49.)? (21, 49.)
row 14: (6, 49.)? (13, 49.)? (14, -196.)? (15, 49.)? (22, 49.)
row 15: (15, 1.)
row 16: (16, 1.)
row 17: (9, 49.)? (16, 49.)? (17, -196.)? (18, 49.)? (25, 49.)
row 18: (10, 49.)? (17, 49.)? (18, -196.)? (19, 49.)? (26, 49.)
row 19: (11, 49.)? (18, 49.)? (19, -196.)? (20, 49.)? (27, 49.)
row 20: (12, 49.)? (19, 49.)? (20, -196.)? (21, 49.)? (28, 49.)
row 21: (13, 49.)? (20, 49.)? (21, -196.)? (22, 49.)? (29, 49.)
row 22: (14, 49.)? (21, 49.)? (22, -196.)? (23, 49.)? (30, 49.)
row 23: (23, 1.)
row 24: (24, 1.)
row 25: (17, 49.)? (24, 49.)? (25, -196.)? (26, 49.)? (33, 49.)
row 26: (18, 49.)? (25, 49.)? (26, -196.)? (27, 49.)? (34, 49.)
row 27: (19, 49.)? (26, 49.)? (27, -196.)? (28, 49.)? (35, 49.)
row 28: (20, 49.)? (27, 49.)? (28, -196.)? (29, 49.)? (36, 49.)
row 29: (21, 49.)? (28, 49.)? (29, -196.)? (30, 49.)? (37, 49.)
row 30: (22, 49.)? (29, 49.)? (30, -196.)? (31, 49.)? (38, 49.)
row 31: (31, 1.)
row 32: (32, 1.)
row 33: (25, 49.)? (32, 49.)? (33, -196.)? (34, 49.)? (41, 49.)
row 34: (26, 49.)? (33, 49.)? (34, -196.)? (35, 49.)? (42, 49.)
row 35: (27, 49.)? (34, 49.)? (35, -196.)? (36, 49.)? (43, 49.)
row 36: (28, 49.)? (35, 49.)? (36, -196.)? (37, 49.)? (44, 49.)
row 37: (29, 49.)? (36, 49.)? (37, -196.)? (38, 49.)? (45, 49.)
row 38: (30, 49.)? (37, 49.)? (38, -196.)? (39, 49.)? (46, 49.)
row 39: (39, 1.)
row 40: (40, 1.)
row 41: (33, 49.)? (40, 49.)? (41, -196.)? (42, 49.)? (49, 49.)
row 42: (34, 49.)? (41, 49.)? (42, -196.)? (43, 49.)? (50, 49.)
row 43: (35, 49.)? (42, 49.)? (43, -196.)? (44, 49.)? (51, 49.)
row 44: (36, 49.)? (43, 49.)? (44, -196.)? (45, 49.)? (52, 49.)
row 45: (37, 49.)? (44, 49.)? (45, -196.)? (46, 49.)? (53, 49.)
row 46: (38, 49.)? (45, 49.)? (46, -196.)? (47, 49.)? (54, 49.)
row 47: (47, 1.)
row 48: (48, 1.)
row 49: (41, 49.)? (48, 49.)? (49, -196.)? (50, 49.)? (57, 49.)
row 50: (42, 49.)? (49, 49.)? (50, -196.)? (51, 49.)? (58, 49.)
row 51: (43, 49.)? (50, 49.)? (51, -196.)? (52, 49.)? (59, 49.)
row 52: (44, 49.)? (51, 49.)? (52, -196.)? (53, 49.)? (60, 49.)
row 53: (45, 49.)? (52, 49.)? (53, -196.)? (54, 49.)? (61, 49.)
row 54: (46, 49.)? (53, 49.)? (54, -196.)? (55, 49.)? (62, 49.)
row 55: (55, 1.)
row 56: (56, 1.)
row 57: (57, 1.)
row 58: (58, 1.)
row 59: (59, 1.)
row 60: (60, 1.)
row 61: (61, 1.)
row 62: (62, 1.)
row 63: (63, 1.)

This the output of the original C ex13 code:

Mat Object: 1 MPI processes
 ? type: seqaij
row 0: (0, 1.)? (1, 0.)? (8, 0.)
row 1: (0, 0.)? (1, 1.)? (2, 0.)? (9, 0.)
row 2: (1, 0.)? (2, 1.)? (3, 0.)? (10, 0.)
row 3: (2, 0.)? (3, 1.)? (4, 0.)? (11, 0.)
row 4: (3, 0.)? (4, 1.)? (5, 0.)? (12, 0.)
row 5: (4, 0.)? (5, 1.)? (6, 0.)? (13, 0.)
row 6: (5, 0.)? (6, 1.)? (7, 0.)? (14, 0.)
row 7: (6, 0.)? (7, 1.)? (15, 0.)
row 8: (0, 0.)? (8, 1.)? (9, 0.)? (16, 0.)
row 9: (1, 49.)? (8, 49.)? (9, -196.)? (10, 49.)? (17, 49.)
row 10: (2, 49.)? (9, 49.)? (10, -196.)? (11, 49.)? (18, 49.)
row 11: (3, 49.)? (10, 49.)? (11, -196.)? (12, 49.)? (19, 49.)
row 12: (4, 49.)? (11, 49.)? (12, -196.)? (13, 49.)? (20, 49.)
row 13: (5, 49.)? (12, 49.)? (13, -196.)? (14, 49.)? (21, 49.)
row 14: (6, 49.)? (13, 49.)? (14, -196.)? (15, 49.)? (22, 49.)
row 15: (7, 0.)? (14, 0.)? (15, 1.)? (23, 0.)
row 16: (8, 0.)? (16, 1.)? (17, 0.)? (24, 0.)
row 17: (9, 49.)? (16, 49.)? (17, -196.)? (18, 49.)? (25, 49.)
row 18: (10, 49.)? (17, 49.)? (18, -196.)? (19, 49.)? (26, 49.)
row 19: (11, 49.)? (18, 49.)? (19, -196.)? (20, 49.)? (27, 49.)
row 20: (12, 49.)? (19, 49.)? (20, -196.)? (21, 49.)? (28, 49.)
row 21: (13, 49.)? (20, 49.)? (21, -196.)? (22, 49.)? (29, 49.)
row 22: (14, 49.)? (21, 49.)? (22, -196.)? (23, 49.)? (30, 49.)
row 23: (15, 0.)? (22, 0.)? (23, 1.)? (31, 0.)
row 24: (16, 0.)? (24, 1.)? (25, 0.)? (32, 0.)
row 25: (17, 49.)? (24, 49.)? (25, -196.)? (26, 49.)? (33, 49.)
row 26: (18, 49.)? (25, 49.)? (26, -196.)? (27, 49.)? (34, 49.)
row 27: (19, 49.)? (26, 49.)? (27, -196.)? (28, 49.)? (35, 49.)
row 28: (20, 49.)? (27, 49.)? (28, -196.)? (29, 49.)? (36, 49.)
row 29: (21, 49.)? (28, 49.)? (29, -196.)? (30, 49.)? (37, 49.)
row 30: (22, 49.)? (29, 49.)? (30, -196.)? (31, 49.)? (38, 49.)
row 31: (23, 0.)? (30, 0.)? (31, 1.)? (39, 0.)
row 32: (24, 0.)? (32, 1.)? (33, 0.)? (40, 0.)
row 33: (25, 49.)? (32, 49.)? (33, -196.)? (34, 49.)? (41, 49.)
row 34: (26, 49.)? (33, 49.)? (34, -196.)? (35, 49.)? (42, 49.)
row 35: (27, 49.)? (34, 49.)? (35, -196.)? (36, 49.)? (43, 49.)
row 36: (28, 49.)? (35, 49.)? (36, -196.)? (37, 49.)? (44, 49.)
row 37: (29, 49.)? (36, 49.)? (37, -196.)? (38, 49.)? (45, 49.)
row 38: (30, 49.)? (37, 49.)? (38, -196.)? (39, 49.)? (46, 49.)
row 39: (31, 0.)? (38, 0.)? (39, 1.)? (47, 0.)
row 40: (32, 0.)? (40, 1.)? (41, 0.)? (48, 0.)
row 41: (33, 49.)? (40, 49.)? (41, -196.)? (42, 49.)? (49, 49.)
row 42: (34, 49.)? (41, 49.)? (42, -196.)? (43, 49.)? (50, 49.)
row 43: (35, 49.)? (42, 49.)? (43, -196.)? (44, 49.)? (51, 49.)
row 44: (36, 49.)? (43, 49.)? (44, -196.)? (45, 49.)? (52, 49.)
row 45: (37, 49.)? (44, 49.)? (45, -196.)? (46, 49.)? (53, 49.)
row 46: (38, 49.)? (45, 49.)? (46, -196.)? (47, 49.)? (54, 49.)
row 47: (39, 0.)? (46, 0.)? (47, 1.)? (55, 0.)
row 48: (40, 0.)? (48, 1.)? (49, 0.)? (56, 0.)
row 49: (41, 49.)? (48, 49.)? (49, -196.)? (50, 49.)? (57, 49.)
row 50: (42, 49.)? (49, 49.)? (50, -196.)? (51, 49.)? (58, 49.)
row 51: (43, 49.)? (50, 49.)? (51, -196.)? (52, 49.)? (59, 49.)
row 52: (44, 49.)? (51, 49.)? (52, -196.)? (53, 49.)? (60, 49.)
row 53: (45, 49.)? (52, 49.)? (53, -196.)? (54, 49.)? (61, 49.)
row 54: (46, 49.)? (53, 49.)? (54, -196.)? (55, 49.)? (62, 49.)
row 55: (47, 0.)? (54, 0.)? (55, 1.)? (63, 0.)
row 56: (48, 0.)? (56, 1.)? (57, 0.)
row 57: (49, 0.)? (56, 0.)? (57, 1.)? (58, 0.)
row 58: (50, 0.)? (57, 0.)? (58, 1.)? (59, 0.)
row 59: (51, 0.)? (58, 0.)? (59, 1.)? (60, 0.)
row 60: (52, 0.)? (59, 0.)? (60, 1.)? (61, 0.)
row 61: (53, 0.)? (60, 0.)? (61, 1.)? (62, 0.)
row 62: (54, 0.)? (61, 0.)? (62, 1.)? (63, 0.)
row 63: (55, 0.)? (62, 0.)? (63, 1.)

I'm filling in the wrong way.
Thanks.

Nicola


-- 

Nicola Creati
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
off.   +39 040 2140 213
fax.   +39 040 327307

_____________________________________________________________________
This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.


From lzou at anl.gov  Mon Apr  1 09:54:46 2019
From: lzou at anl.gov (Zou, Ling)
Date: Mon, 1 Apr 2019 14:54:46 +0000
Subject: [petsc-users] Problem about Residual evaluation
In-Reply-To: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
References: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
Message-ID: <2A923344-3DDC-490B-85BB-E3EFE1CE85CB@anl.gov>

For question 2, there are some discussions in PETSc?s manual section 4.3 and 4.3.3.
https://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf

-Ling

From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov>
Reply-To: Yingjie Wu <yjwu16 at gmail.com>
Date: Monday, April 1, 2019 at 9:22 AM
To: "petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
Subject: [petsc-users] Problem about Residual evaluation

Dear PETSc developers:
Hi,

I've been using -snes_mf_operator and I've customized a precondition matrix to solve my problem.I have two questions about the residuals of linear steps(KSP residual).



1.Since I'm using a matrix-free method, how do we get KSP residuals in PETSc?

r_m = b - A*x_m

Is finite difference used to approximate "A*x_m" ?



2.What is the difference between instruction ' -ksp_monitor ' and ' -ksp_monitor_true_residual ' in how they are calculated?



Thanks,

Yingjie




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/3737220c/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr  1 09:59:01 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 1 Apr 2019 14:59:01 +0000
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <6e05c686-b3a7-7015-ff27-9772881979ef@inogs.it>
References: <6e05c686-b3a7-7015-ff27-9772881979ef@inogs.it>
Message-ID: <D6F1724C-618E-44F7-88C2-648AB80CE46F@anl.gov>


   

> On Apr 1, 2019, at 9:45 AM, Nicola Creati via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thanks Matt, it solves the problem. The Jacobian matrix is now filled but checking between the C and Python code with MatView there are some differences.
> This is my Jacobian matrix:

   Hmm, just doing a quick eye-ball scan it looks like the matrices have the same numerical values; it is just that the C version has also inserted some additional 0 values at other locations. The solution to both linear systems should be the same but note that the iterative method used to solve the system may converge differently (because some of the preconditions in PETSc construct the preconditioner based on the "non-zero structure" of the matrix and in this construction the 0 locations can affect the preconditioner) hence the partial solution after running the solver may be slightly different between the two solvers. If you run with a direct solver -pc_type lu you should get two answers that are very close to each other.

    Barry

> 
> Mat Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 1.)
> row 1: (1, 1.)
> row 2: (2, 1.)
> row 3: (3, 1.)
> row 4: (4, 1.)
> row 5: (5, 1.)
> row 6: (6, 1.)
> row 7: (7, 1.)
> row 8: (8, 1.)
> row 9: (1, 49.)  (8, 49.)  (9, -196.)  (10, 49.)  (17, 49.)
> row 10: (2, 49.)  (9, 49.)  (10, -196.)  (11, 49.)  (18, 49.)
> row 11: (3, 49.)  (10, 49.)  (11, -196.)  (12, 49.)  (19, 49.)
> row 12: (4, 49.)  (11, 49.)  (12, -196.)  (13, 49.)  (20, 49.)
> row 13: (5, 49.)  (12, 49.)  (13, -196.)  (14, 49.)  (21, 49.)
> row 14: (6, 49.)  (13, 49.)  (14, -196.)  (15, 49.)  (22, 49.)
> row 15: (15, 1.)
> row 16: (16, 1.)
> row 17: (9, 49.)  (16, 49.)  (17, -196.)  (18, 49.)  (25, 49.)
> row 18: (10, 49.)  (17, 49.)  (18, -196.)  (19, 49.)  (26, 49.)
> row 19: (11, 49.)  (18, 49.)  (19, -196.)  (20, 49.)  (27, 49.)
> row 20: (12, 49.)  (19, 49.)  (20, -196.)  (21, 49.)  (28, 49.)
> row 21: (13, 49.)  (20, 49.)  (21, -196.)  (22, 49.)  (29, 49.)
> row 22: (14, 49.)  (21, 49.)  (22, -196.)  (23, 49.)  (30, 49.)
> row 23: (23, 1.)
> row 24: (24, 1.)
> row 25: (17, 49.)  (24, 49.)  (25, -196.)  (26, 49.)  (33, 49.)
> row 26: (18, 49.)  (25, 49.)  (26, -196.)  (27, 49.)  (34, 49.)
> row 27: (19, 49.)  (26, 49.)  (27, -196.)  (28, 49.)  (35, 49.)
> row 28: (20, 49.)  (27, 49.)  (28, -196.)  (29, 49.)  (36, 49.)
> row 29: (21, 49.)  (28, 49.)  (29, -196.)  (30, 49.)  (37, 49.)
> row 30: (22, 49.)  (29, 49.)  (30, -196.)  (31, 49.)  (38, 49.)
> row 31: (31, 1.)
> row 32: (32, 1.)
> row 33: (25, 49.)  (32, 49.)  (33, -196.)  (34, 49.)  (41, 49.)
> row 34: (26, 49.)  (33, 49.)  (34, -196.)  (35, 49.)  (42, 49.)
> row 35: (27, 49.)  (34, 49.)  (35, -196.)  (36, 49.)  (43, 49.)
> row 36: (28, 49.)  (35, 49.)  (36, -196.)  (37, 49.)  (44, 49.)
> row 37: (29, 49.)  (36, 49.)  (37, -196.)  (38, 49.)  (45, 49.)
> row 38: (30, 49.)  (37, 49.)  (38, -196.)  (39, 49.)  (46, 49.)
> row 39: (39, 1.)
> row 40: (40, 1.)
> row 41: (33, 49.)  (40, 49.)  (41, -196.)  (42, 49.)  (49, 49.)
> row 42: (34, 49.)  (41, 49.)  (42, -196.)  (43, 49.)  (50, 49.)
> row 43: (35, 49.)  (42, 49.)  (43, -196.)  (44, 49.)  (51, 49.)
> row 44: (36, 49.)  (43, 49.)  (44, -196.)  (45, 49.)  (52, 49.)
> row 45: (37, 49.)  (44, 49.)  (45, -196.)  (46, 49.)  (53, 49.)
> row 46: (38, 49.)  (45, 49.)  (46, -196.)  (47, 49.)  (54, 49.)
> row 47: (47, 1.)
> row 48: (48, 1.)
> row 49: (41, 49.)  (48, 49.)  (49, -196.)  (50, 49.)  (57, 49.)
> row 50: (42, 49.)  (49, 49.)  (50, -196.)  (51, 49.)  (58, 49.)
> row 51: (43, 49.)  (50, 49.)  (51, -196.)  (52, 49.)  (59, 49.)
> row 52: (44, 49.)  (51, 49.)  (52, -196.)  (53, 49.)  (60, 49.)
> row 53: (45, 49.)  (52, 49.)  (53, -196.)  (54, 49.)  (61, 49.)
> row 54: (46, 49.)  (53, 49.)  (54, -196.)  (55, 49.)  (62, 49.)
> row 55: (55, 1.)
> row 56: (56, 1.)
> row 57: (57, 1.)
> row 58: (58, 1.)
> row 59: (59, 1.)
> row 60: (60, 1.)
> row 61: (61, 1.)
> row 62: (62, 1.)
> row 63: (63, 1.)
> 
> This the output of the original C ex13 code:
> 
> Mat Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 1.)  (1, 0.)  (8, 0.)
> row 1: (0, 0.)  (1, 1.)  (2, 0.)  (9, 0.)
> row 2: (1, 0.)  (2, 1.)  (3, 0.)  (10, 0.)
> row 3: (2, 0.)  (3, 1.)  (4, 0.)  (11, 0.)
> row 4: (3, 0.)  (4, 1.)  (5, 0.)  (12, 0.)
> row 5: (4, 0.)  (5, 1.)  (6, 0.)  (13, 0.)
> row 6: (5, 0.)  (6, 1.)  (7, 0.)  (14, 0.)
> row 7: (6, 0.)  (7, 1.)  (15, 0.)
> row 8: (0, 0.)  (8, 1.)  (9, 0.)  (16, 0.)
> row 9: (1, 49.)  (8, 49.)  (9, -196.)  (10, 49.)  (17, 49.)
> row 10: (2, 49.)  (9, 49.)  (10, -196.)  (11, 49.)  (18, 49.)
> row 11: (3, 49.)  (10, 49.)  (11, -196.)  (12, 49.)  (19, 49.)
> row 12: (4, 49.)  (11, 49.)  (12, -196.)  (13, 49.)  (20, 49.)
> row 13: (5, 49.)  (12, 49.)  (13, -196.)  (14, 49.)  (21, 49.)
> row 14: (6, 49.)  (13, 49.)  (14, -196.)  (15, 49.)  (22, 49.)
> row 15: (7, 0.)  (14, 0.)  (15, 1.)  (23, 0.)
> row 16: (8, 0.)  (16, 1.)  (17, 0.)  (24, 0.)
> row 17: (9, 49.)  (16, 49.)  (17, -196.)  (18, 49.)  (25, 49.)
> row 18: (10, 49.)  (17, 49.)  (18, -196.)  (19, 49.)  (26, 49.)
> row 19: (11, 49.)  (18, 49.)  (19, -196.)  (20, 49.)  (27, 49.)
> row 20: (12, 49.)  (19, 49.)  (20, -196.)  (21, 49.)  (28, 49.)
> row 21: (13, 49.)  (20, 49.)  (21, -196.)  (22, 49.)  (29, 49.)
> row 22: (14, 49.)  (21, 49.)  (22, -196.)  (23, 49.)  (30, 49.)
> row 23: (15, 0.)  (22, 0.)  (23, 1.)  (31, 0.)
> row 24: (16, 0.)  (24, 1.)  (25, 0.)  (32, 0.)
> row 25: (17, 49.)  (24, 49.)  (25, -196.)  (26, 49.)  (33, 49.)
> row 26: (18, 49.)  (25, 49.)  (26, -196.)  (27, 49.)  (34, 49.)
> row 27: (19, 49.)  (26, 49.)  (27, -196.)  (28, 49.)  (35, 49.)
> row 28: (20, 49.)  (27, 49.)  (28, -196.)  (29, 49.)  (36, 49.)
> row 29: (21, 49.)  (28, 49.)  (29, -196.)  (30, 49.)  (37, 49.)
> row 30: (22, 49.)  (29, 49.)  (30, -196.)  (31, 49.)  (38, 49.)
> row 31: (23, 0.)  (30, 0.)  (31, 1.)  (39, 0.)
> row 32: (24, 0.)  (32, 1.)  (33, 0.)  (40, 0.)
> row 33: (25, 49.)  (32, 49.)  (33, -196.)  (34, 49.)  (41, 49.)
> row 34: (26, 49.)  (33, 49.)  (34, -196.)  (35, 49.)  (42, 49.)
> row 35: (27, 49.)  (34, 49.)  (35, -196.)  (36, 49.)  (43, 49.)
> row 36: (28, 49.)  (35, 49.)  (36, -196.)  (37, 49.)  (44, 49.)
> row 37: (29, 49.)  (36, 49.)  (37, -196.)  (38, 49.)  (45, 49.)
> row 38: (30, 49.)  (37, 49.)  (38, -196.)  (39, 49.)  (46, 49.)
> row 39: (31, 0.)  (38, 0.)  (39, 1.)  (47, 0.)
> row 40: (32, 0.)  (40, 1.)  (41, 0.)  (48, 0.)
> row 41: (33, 49.)  (40, 49.)  (41, -196.)  (42, 49.)  (49, 49.)
> row 42: (34, 49.)  (41, 49.)  (42, -196.)  (43, 49.)  (50, 49.)
> row 43: (35, 49.)  (42, 49.)  (43, -196.)  (44, 49.)  (51, 49.)
> row 44: (36, 49.)  (43, 49.)  (44, -196.)  (45, 49.)  (52, 49.)
> row 45: (37, 49.)  (44, 49.)  (45, -196.)  (46, 49.)  (53, 49.)
> row 46: (38, 49.)  (45, 49.)  (46, -196.)  (47, 49.)  (54, 49.)
> row 47: (39, 0.)  (46, 0.)  (47, 1.)  (55, 0.)
> row 48: (40, 0.)  (48, 1.)  (49, 0.)  (56, 0.)
> row 49: (41, 49.)  (48, 49.)  (49, -196.)  (50, 49.)  (57, 49.)
> row 50: (42, 49.)  (49, 49.)  (50, -196.)  (51, 49.)  (58, 49.)
> row 51: (43, 49.)  (50, 49.)  (51, -196.)  (52, 49.)  (59, 49.)
> row 52: (44, 49.)  (51, 49.)  (52, -196.)  (53, 49.)  (60, 49.)
> row 53: (45, 49.)  (52, 49.)  (53, -196.)  (54, 49.)  (61, 49.)
> row 54: (46, 49.)  (53, 49.)  (54, -196.)  (55, 49.)  (62, 49.)
> row 55: (47, 0.)  (54, 0.)  (55, 1.)  (63, 0.)
> row 56: (48, 0.)  (56, 1.)  (57, 0.)
> row 57: (49, 0.)  (56, 0.)  (57, 1.)  (58, 0.)
> row 58: (50, 0.)  (57, 0.)  (58, 1.)  (59, 0.)
> row 59: (51, 0.)  (58, 0.)  (59, 1.)  (60, 0.)
> row 60: (52, 0.)  (59, 0.)  (60, 1.)  (61, 0.)
> row 61: (53, 0.)  (60, 0.)  (61, 1.)  (62, 0.)
> row 62: (54, 0.)  (61, 0.)  (62, 1.)  (63, 0.)
> row 63: (55, 0.)  (62, 0.)  (63, 1.)
> 
> I'm filling in the wrong way.
> Thanks.
> 
> Nicola
> 
> 
> -- 
> 
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
> CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
> 
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.
> 


From ncreati at inogs.it  Mon Apr  1 10:28:02 2019
From: ncreati at inogs.it (Nicola Creati)
Date: Mon, 1 Apr 2019 17:28:02 +0200
Subject: [petsc-users] petsc4py - Convert c example code to Python
Message-ID: <7dc9f7fb-5648-cc98-cd65-f83ec6209d4b@inogs.it>

The Python and C solutions are different. I used the following commands:

Python: python ex13.py -ts_max_step 5 -pc_type lu

C: ./ex13 -ts_max_step 5 -pc_type lu

I attached the two solutions, values and distributions are different.

Thanks.

Nicola

-- 

Nicola Creati
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
off.   +39 040 2140 213
fax.   +39 040 327307

_____________________________________________________________________
This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: python_sol.png
Type: image/png
Size: 12219 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/d59477d9/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: c_sol.png
Type: image/png
Size: 13695 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/d59477d9/attachment-0003.png>

From bsmith at mcs.anl.gov  Mon Apr  1 10:36:30 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 1 Apr 2019 15:36:30 +0000
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <7dc9f7fb-5648-cc98-cd65-f83ec6209d4b@inogs.it>
References: <7dc9f7fb-5648-cc98-cd65-f83ec6209d4b@inogs.it>
Message-ID: <9105AB48-6CB5-47B9-898D-41B85EDA7210@anl.gov>

 You are in a very lucky position in terms of debugging because you have a running code you can compare with, take advantage of this fact.

   Ok, you're going to have to see when the two codes start diverging from each other. First run with 

-ts_monitor -snes_monitor -ksp_monitor 

Do the initial values for the monitor match, or do no values match? 

  Put at VecView in both codes on your initial value for x (the initial conditions for the ODE) are they the same?

  Next you should put a VecView() at the end of your IFunction routine and a MatView at the end of your IJacobian routine to make sure they are producing the same values.

 Good luck

   Barry



> On Apr 1, 2019, at 10:28 AM, Nicola Creati via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> The Python and C solutions are different. I used the following commands:
> 
> Python: python ex13.py -ts_max_step 5 -pc_type lu
> 
> C: ./ex13 -ts_max_step 5 -pc_type lu
> 
> I attached the two solutions, values and distributions are different.
> 
> Thanks.
> 
> Nicola
> 
> -- 
> 
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
> CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
> 
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.
> 
> <python_sol.png><c_sol.png>


From knepley at gmail.com  Mon Apr  1 11:22:40 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Apr 2019 12:22:40 -0400
Subject: [petsc-users] Problem about Residual evaluation
In-Reply-To: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
References: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
Message-ID: <CAMYG4GmpLzqZZV-MbGaeAJUwBfBxPy2P8+b5_QkkR-YhbtwVOQ@mail.gmail.com>

On Mon, Apr 1, 2019 at 10:22 AM Yingjie Wu via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear PETSc developers:
> Hi,
>
> I've been using -snes_mf_operator and I've customized a precondition
> matrix to solve my problem.I have two questions about the residuals of
> linear steps(KSP residual).
>
>
> 1.Since I'm using a matrix-free method, how do we get KSP residuals in
> PETSc?
>
> r_m = b - A*x_m
>
> Is finite difference used to approximate "A*x_m" ?
>

Yes.


> 2.What is the difference between instruction ' -ksp_monitor ' and '
> -ksp_monitor_true_residual ' in how they are calculated?
>

The true residual is the unpreconditioned residual.

  Matt


> Thanks,
>
> Yingjie
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/31a9b5fa/attachment.html>

From fdkong.jd at gmail.com  Mon Apr  1 11:36:05 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Mon, 1 Apr 2019 10:36:05 -0600
Subject: [petsc-users] Problem about Residual evaluation
In-Reply-To: <CAMYG4GmpLzqZZV-MbGaeAJUwBfBxPy2P8+b5_QkkR-YhbtwVOQ@mail.gmail.com>
References: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
	<CAMYG4GmpLzqZZV-MbGaeAJUwBfBxPy2P8+b5_QkkR-YhbtwVOQ@mail.gmail.com>
Message-ID: <CAN5Wd-LDGe6Qop9amwocmVUmKCvyTeas5DM+5ZCRi=+tWX7DLQ@mail.gmail.com>

On Mon, Apr 1, 2019 at 10:24 AM Matthew Knepley via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> On Mon, Apr 1, 2019 at 10:22 AM Yingjie Wu via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear PETSc developers:
>> Hi,
>>
>> I've been using -snes_mf_operator and I've customized a precondition
>> matrix to solve my problem.I have two questions about the residuals of
>> linear steps(KSP residual).
>>
>>
>> 1.Since I'm using a matrix-free method, how do we get KSP residuals in
>> PETSc?
>>
>> r_m = b - A*x_m
>>
>> Is finite difference used to approximate "A*x_m" ?
>>
>
> Yes.
>
>
>> 2.What is the difference between instruction ' -ksp_monitor ' and '
>> -ksp_monitor_true_residual ' in how they are calculated?
>>
>
> The true residual is the unpreconditioned residual.
>

I actually have some specific understanding on " -ksp_monitor_true_residual",
but not sure it is right or not.  If I am wrong, please correct me.

When the preconditioning  matrix is super ill-conditioned, the ``true
residual" is not necessary ``true"  for the right preconditioning since an
unwind process is applied.  That is,   "-ksp_monitor_true_residual" does
not print ||b-Ax||, instead, it prints  the unpreconditioned residual by
unpreconditioning  the preconditioned residual.


Thanks,

Fande,



>
>   Matt
>
>
>> Thanks,
>>
>> Yingjie
>>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/760fc79a/attachment.html>

From knepley at gmail.com  Mon Apr  1 11:46:27 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Apr 2019 12:46:27 -0400
Subject: [petsc-users] Problem about Residual evaluation
In-Reply-To: <CAN5Wd-LDGe6Qop9amwocmVUmKCvyTeas5DM+5ZCRi=+tWX7DLQ@mail.gmail.com>
References: <CAAMEhBj1Rjxoweg6TrBOX=AiqUNjLOKQeicQ7WPgWvNjsw+Qew@mail.gmail.com>
	<CAMYG4GmpLzqZZV-MbGaeAJUwBfBxPy2P8+b5_QkkR-YhbtwVOQ@mail.gmail.com>
	<CAN5Wd-LDGe6Qop9amwocmVUmKCvyTeas5DM+5ZCRi=+tWX7DLQ@mail.gmail.com>
Message-ID: <CAMYG4G=vg3nnTV+Yzf9EO9DSkVChCNNFnFWB+3ceD2yk0qz_Eg@mail.gmail.com>

On Mon, Apr 1, 2019 at 12:36 PM Fande Kong <fdkong.jd at gmail.com> wrote:

> On Mon, Apr 1, 2019 at 10:24 AM Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> On Mon, Apr 1, 2019 at 10:22 AM Yingjie Wu via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Dear PETSc developers:
>>> Hi,
>>>
>>> I've been using -snes_mf_operator and I've customized a precondition
>>> matrix to solve my problem.I have two questions about the residuals of
>>> linear steps(KSP residual).
>>>
>>>
>>> 1.Since I'm using a matrix-free method, how do we get KSP residuals in
>>> PETSc?
>>>
>>> r_m = b - A*x_m
>>>
>>> Is finite difference used to approximate "A*x_m" ?
>>>
>>
>> Yes.
>>
>>
>>> 2.What is the difference between instruction ' -ksp_monitor ' and '
>>> -ksp_monitor_true_residual ' in how they are calculated?
>>>
>>
>> The true residual is the unpreconditioned residual.
>>
>
> I actually have some specific understanding on "
> -ksp_monitor_true_residual", but not sure it is right or not.  If I am
> wrong, please correct me.
>
> When the preconditioning  matrix is super ill-conditioned, the ``true
> residual" is not necessary ``true"  for the right preconditioning since an
> unwind process is applied.  That is,   "-ksp_monitor_true_residual" does
> not print ||b-Ax||, instead, it prints  the unpreconditioned residual by
> unpreconditioning  the preconditioned residual.
>

This calls KSPBuildResidual(). Everything but GCR calls
KSPBuildResidualDefault(). This just computes b - Ax.
What you may be talking about is how you get x when you are right
preconditioning. This calls KSPBuildSolution().
With right preconditioning, you must apply the preconditioner to get back
to the original space, but this IS EXACTLY
the true residual.

  Thanks,

    Matt


> Thanks,
>
> Fande,
>
>
>
>>
>>   Matt
>>
>>
>>> Thanks,
>>>
>>> Yingjie
>>>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190401/ffe3ef8d/attachment.html>

From ncreati at inogs.it  Tue Apr  2 03:18:05 2019
From: ncreati at inogs.it (Nicola Creati)
Date: Tue, 2 Apr 2019 10:18:05 +0200
Subject: [petsc-users] petsc4py - Convert c example code to Python
Message-ID: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>

Hello, I have run the two codes using: -pc_type lu -ts_monitor 
-snes_monitor -ksp_monitor -ts_max_steps 5. The codes start to diverge 
after the first time step:
Python run :

0 TS dt 0.01 time 0.
 ??? 0 SNES Function norm 2.327405179696e+02
 ????? 0 KSP Residual norm 1.939100379240e+00
 ????? 1 KSP Residual norm 1.013760235597e-15
 ??? 1 SNES Function norm 9.538686413612e-14
1 TS dt 0.01 time 0.01
 ??? 0 SNES Function norm 8.565210784140e-14
 ????? 0 KSP Residual norm 1.266678408555e-15
 ????? 1 KSP Residual norm 2.663404219322e-31
 ??? 1 SNES Function norm 5.528402779439e-29
2 TS dt 0.01 time 0.02
 ??? 0 SNES Function norm 5.829656072531e-29
 ????? 0 KSP Residual norm 3.805606380832e-31
 ????? 1 KSP Residual norm 3.283104975114e-46
 ??? 1 SNES Function norm 1.867771204856e-44
3 TS dt 0.01 time 0.03
 ??? 0 SNES Function norm 1.644541571708e-44
 ????? 0 KSP Residual norm 1.969573456719e-46
 ????? 1 KSP Residual norm 1.297976290541e-61
 ??? 1 SNES Function norm 1.780215821230e-59
4 TS dt 0.01 time 0.04
 ??? 0 SNES Function norm 1.700894451833e-59
5 TS dt 0.01 time 0.05

ex13.c run:

0 TS dt 0.01 time 0.
 ??? 0 SNES Function norm 2.327405179696e+02
 ????? 0 KSP Residual norm 9.158498731719e-01
 ????? 1 KSP Residual norm 5.129005976821e-16
 ??? 1 SNES Function norm 1.141297389434e-13
1 TS dt 0.01 time 0.01
 ??? 0 SNES Function norm 9.158498731719e+01
 ????? 0 KSP Residual norm 4.156347391374e-01
 ????? 1 KSP Residual norm 1.996081836894e-16
 ??? 1 SNES Function norm 3.609761734594e-14
2 TS dt 0.01 time 0.02
 ??? 0 SNES Function norm 4.156347391374e+01
 ????? 0 KSP Residual norm 2.203298795311e-01
 ????? 1 KSP Residual norm 9.047356740098e-17
 ??? 1 SNES Function norm 3.292648977280e-14
3 TS dt 0.01 time 0.03
 ??? 0 SNES Function norm 2.203298795311e+01
 ????? 0 KSP Residual norm 1.367458644503e-01
 ????? 1 KSP Residual norm 5.055030661919e-17
 ??? 1 SNES Function norm 1.272304808486e-14
4 TS dt 0.01 time 0.04
 ??? 0 SNES Function norm 1.367458644503e+01
 ????? 0 KSP Residual norm 9.695393464180e-02
 ????? 1 KSP Residual norm 1.918401734795e-17
 ??? 1 SNES Function norm 1.130247628824e-14
5 TS dt 0.01 time 0.05

I rechecked the Python code against the C one but I did not find the 
source of the issue.

# Example 13 petsc TS
import sys, petsc4py
petsc4py.init(sys.argv)

from petsc4py import PETSc
from mpi4py import MPI
import numpy as np
import math

def RHS_func(ts, t, X, xdot, F):
 ??? da = ts.getDM()
 ??? localU = da.getLocalVec()

 ??? da.globalToLocal(X, localU)

 ??? mx, my = da.getSizes()

 ??? hx, hy = [1.0/(m-1) for m in [mx, my]]
 ??? sx = 1.0/(hx*hx)
 ??? sy = 1.0/(hy*hy)

 ??? uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
 ??? f = F.getArray(readonly=0).reshape(8, 8, order='C')

 ??? (xs, xm), (ys, ym) = da.getRanges()
 ??? for j in range(ys, ym):
 ??????? for i in range(xs, xm):
 ??????????? if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
 ??????????????? f[i,j] = uarray[i,j]
 ??????????????? continue
 ??????????? u = uarray[i,j]
 ??????????? uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
 ??????????? uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
 ??????????? f[i, j] = uxx + uyy
 ??? F.assemble()
 ??? #PETSc.Log.logFlops(11.0*xm*ym)


def Jacobian_func(ts, t, x, xdot, a, A, B):
 ??? mx, my = da.getSizes()
 ??? hx = 1.0/(mx-1)
 ??? hy = 1.0/(my-1)
 ??? sx = 1.0/(hx*hx)
 ??? sy = 1.0/(hy*hy)

 ??? B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
 ??? B.zeroEntries()

 ??? (i0, i1), (j0, j1) = da.getRanges()
 ??? row = PETSc.Mat.Stencil()
 ??? col = PETSc.Mat.Stencil()

 ??? for i in range(j0, j1):
 ??????? for j in range(i0, i1):
 ??????????? row.index = (i,j)
 ??????????? row.field = 0
 ??????????? if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
 ??????????????? B.setValueStencil(row, row, 1.0)
 ??????????? else:
 ??????????????? for index, value in [
 ??????????????????? ((i-1, j),?? sx),
 ??????????????????? ((i+1, j),?? sx),
 ??????????????????? ((i,?? j-1), sy),
 ??????????????????? ((i, j+1), sy),
 ??????????????????? ((i,?? j),? -2*sx - 2*sy)]:
 ??????????????????? col.index = index
 ??????????????????? col.field = 0
 ??????????????????? B.setValueStencil(row, col, value)

 ??? B.assemble()
 ??? #B.view()
 ??? if A != B: B.assemble()

 ??? return PETSc.Mat.Structure.SAME_NONZERO_PATTERN

def make_initial_solution(da, U, c):
 ??? mx, my = da.getSizes()
 ??? hx = 1.0/(mx-1)
 ??? hy = 1.0/(my-1)
 ??? (xs, xm), (ys, ym) = da.getRanges()

 ??? u = U.getArray(readonly=0).reshape(8, 8, order='C')

 ??? for j in range(ys, ym):
 ??????? y = j*hy
 ??????? for i in range(xs, xm):
 ??????????? x = i*hx
 ??????????? r = math.sqrt((x-0.5)**2+(y-0.5)**2)
 ??????????? if r < (0.125):
 ??????????????? u[i, j] = math.exp(c*r*r*r)
 ??????????? else:
 ??????????????? u[i, j] = 0.0
 ??? U.assemble()
 ??? #U.view()


def monitor(ts, i, t, x):
 ??? xx = x[:].tolist()
 ??? history.append((i, t, xx))


history = []
nx = 8
ny = 8
da = PETSc.DMDA().create([nx, ny], stencil_type= 
PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)

da.setFromOptions()
da.setUp()

u = da.createGlobalVec()
f = u.duplicate()

c = -30.0

ts = PETSc.TS().create()
ts.setDM(da)
ts.setType(ts.Type.BEULER)

ts.setIFunction(RHS_func, f)

da.setMatType('aij')
J = da.createMat()
ts.setIJacobian(Jacobian_func, J, J)

ftime = 1.0
ts.setMaxTime(ftime)
ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)

make_initial_solution(da, u, c)
dt = 0.01

ts.setMonitor(monitor)
ts.setTimeStep(dt)
ts.setFromOptions()
ts.solve(u)

ftime = ts.getSolveTime()
steps = ts.getStepNumber()

Thanks.

Nicola


-- 

Nicola Creati
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
off.   +39 040 2140 213
fax.   +39 040 327307

_____________________________________________________________________
This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.


From eda.oktay at metu.edu.tr  Tue Apr  2 05:54:38 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 2 Apr 2019 13:54:38 +0300
Subject: [petsc-users] Local and global size of IS
In-Reply-To: <202187C0-0CD5-46F6-BE73-4F48243BE119@anl.gov>
References: <CA+vu_Xh6Qr8c=mrkDccbxBNgd3KDO8B92S7Z_SRTXFcby+43BQ@mail.gmail.com>
	<202187C0-0CD5-46F6-BE73-4F48243BE119@anl.gov>
Message-ID: <CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>

Hi Barry,

I did what you said but now I get the error in ISSetPermutation that index
set is not a permutation. ? think it is because since I selected indices on
each process as you said. I did the following, did I understand you wrong:

  PetscMPIInt rank,size;
  MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
  MPI_Comm_size(PETSC_COMM_WORLD, &size);

  PetscInt *idxx,ss;
  ss = siz/size;
  PetscMalloc1(ss,&idxx);
  if (rank != size-1) {
    j =0;
    for (i=rank*ss; i<(rank+1)*ss; i++) {
      idxx[j] = idx[i];
      j++;
    }

  } else {

    j =0;
    for (i=rank*ss; i<siz; i++) {
      idxx[j] = idx[i];
      j++;
    }

  }

Smith, Barry F. <bsmith at mcs.anl.gov>, 31 Mar 2019 Paz, 01:47 tarihinde ?unu
yazd?:

>
>
> > On Mar 27, 2019, at 7:33 AM, Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello,
> >
> > I am trying to permute a matrix A(of size 2n*2n) according to a sorted
> eigenvector vr (of size 2n) in parallel using 2 processors (processor
> number can change).
>
>     I don't understand your rational for wanting to permute the matrix
> rows/columns based on a sorted eigenvector?
>
> > However, I get an error in MatPermute line stating that arguments are
> out of range and a new nonzero caused a malloc even if I used MatSetOption.
> >
> > I discovered that this may be because of the unequality of local sizes
> of is (and newIS) and local size of A.
> >
> > Since I allocate index set idx according to size of the vector vr,
> global size of is becomes 2n and the local size is also 2n (I think it
> should be n since both A and vr has local sizes n because of the number of
> processors). If I change the size of idx, then because of VecGetArrayRead,
> I think is is created wrongly.
> >
> > So, how can I make both global and local sizes of is,newIS and A?
> >
> > Below, you can see the part of my program.
> >
> > Thanks,
> >
> > Eda
> >
> >  ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
> >   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
> >   for (i=0; i<siz;i++) idx[i] = i;
>
>    VecGetArray only gives you access to the local portion of the vector,
> not the entire vector so you cannot do the next two lines of code
> >   ierr = VecGetArrayRead(vr,&avr);CHKERRQ(ierr);
>
>    The sort routine is sequential; it knows nothing about parallelism so
> each process is just sorting its part. Hence this code won't work as
> expected
>
> >   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>
>    If you need to sort the parallel vector and get the entire permutation
> then you need to do the following
>
> 1)   communicate the entire vector to each process, you can use
> VecScatterCreateToAll() for this
> 2)   call VecGetArray on the new vector (that contains all entries on each
> process)
> 3)   Call PetscSortRealWithPermutation() on the entire vector on each
> process
> 4)   select out a piece of the resulting indices idx on each process; for
> example with two processes I think rank = 0 would get the first half of the
> idx and rank = 1 would get the second half.
> >
> >   ierr =
> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>
> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>
>   You don't need to duplicate the is, just pass it in twice.
>
> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>
>    You should definitely not need the next line. The A matrix is untouched
> by the subroutine call so you don't need to change its allocation options
>
> >
> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/f246a49b/attachment.html>

From mfadams at lbl.gov  Tue Apr  2 07:01:41 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 2 Apr 2019 08:01:41 -0400
Subject: [petsc-users] Local and global size of IS
In-Reply-To: <CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>
References: <CA+vu_Xh6Qr8c=mrkDccbxBNgd3KDO8B92S7Z_SRTXFcby+43BQ@mail.gmail.com>
	<202187C0-0CD5-46F6-BE73-4F48243BE119@anl.gov>
	<CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>
Message-ID: <CADOhEh64Gy-skGMt-fd1eRF4T76++SttXkvd8cuwfWROKO2Z=g@mail.gmail.com>

On Tue, Apr 2, 2019 at 6:55 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi Barry,
>
> I did what you said but now I get the error in ISSetPermutation that index
> set is not a permutation. ? think it is because since I selected indices on
> each process as you said. I did the following, did I understand you wrong:
>
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>
>   PetscInt *idxx,ss;
>   ss = siz/size;
>   PetscMalloc1(ss,&idxx);
>   if (rank != size-1) {
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>
>   } else {
>
>     j =0;
>     for (i=rank*ss; i<siz; i++) {
>       idxx[j] = idx[i];
>

if siz%size != 0, you will run off the end of idxx, that is you have a
memory error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/e7be1c64/attachment.html>

From eda.oktay at metu.edu.tr  Tue Apr  2 08:00:13 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 2 Apr 2019 16:00:13 +0300
Subject: [petsc-users] Local and global size of IS
In-Reply-To: <CADOhEh64Gy-skGMt-fd1eRF4T76++SttXkvd8cuwfWROKO2Z=g@mail.gmail.com>
References: <CA+vu_Xh6Qr8c=mrkDccbxBNgd3KDO8B92S7Z_SRTXFcby+43BQ@mail.gmail.com>
	<202187C0-0CD5-46F6-BE73-4F48243BE119@anl.gov>
	<CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>
	<CADOhEh64Gy-skGMt-fd1eRF4T76++SttXkvd8cuwfWROKO2Z=g@mail.gmail.com>
Message-ID: <CA+vu_Xj6_p7f+7Q7W4zMNA_fp5GPJnMA99tjXBse6w0NeC3+Dg@mail.gmail.com>

Hi Mark,

siz%size = 0 in my case because siz=362 and I use at most 2 processors.

Mark Adams <mfadams at lbl.gov>, 2 Nis 2019 Sal, 15:12 tarihinde ?unu yazd?:

>
>
> On Tue, Apr 2, 2019 at 6:55 AM Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi Barry,
>>
>> I did what you said but now I get the error in ISSetPermutation that
>> index set is not a permutation. ? think it is because since I selected
>> indices on each process as you said. I did the following, did I understand
>> you wrong:
>>
>>   PetscMPIInt rank,size;
>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>
>>   PetscInt *idxx,ss;
>>   ss = siz/size;
>>   PetscMalloc1(ss,&idxx);
>>   if (rank != size-1) {
>>     j =0;
>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>       idxx[j] = idx[i];
>>       j++;
>>     }
>>
>>   } else {
>>
>>     j =0;
>>     for (i=rank*ss; i<siz; i++) {
>>       idxx[j] = idx[i];
>>
>
> if siz%size != 0, you will run off the end of idxx, that is you have a
> memory error.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/d961da36/attachment.html>

From knepley at gmail.com  Tue Apr  2 09:07:11 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Apr 2019 10:07:11 -0400
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
References: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
Message-ID: <CAMYG4Gk9V7oCrRopNo=UMyZ+acJ7hqdx8-L05=Tms=Ce1q_Cpg@mail.gmail.com>

On Tue, Apr 2, 2019 at 4:19 AM Nicola Creati via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello, I have run the two codes using: -pc_type lu -ts_monitor
> -snes_monitor -ksp_monitor -ts_max_steps 5. The codes start to diverge
> after the first time step:
>

In RHS_Function, you never use xdot, but this is an implicit TS. Thus, you
are solving the Laplace
equation, not the heat equation, and you get the exact answer after one
step.

  Matt


> Python run :
>
> 0 TS dt 0.01 time 0.
>      0 SNES Function norm 2.327405179696e+02
>        0 KSP Residual norm 1.939100379240e+00
>        1 KSP Residual norm 1.013760235597e-15
>      1 SNES Function norm 9.538686413612e-14
> 1 TS dt 0.01 time 0.01
>      0 SNES Function norm 8.565210784140e-14
>        0 KSP Residual norm 1.266678408555e-15
>        1 KSP Residual norm 2.663404219322e-31
>      1 SNES Function norm 5.528402779439e-29
> 2 TS dt 0.01 time 0.02
>      0 SNES Function norm 5.829656072531e-29
>        0 KSP Residual norm 3.805606380832e-31
>        1 KSP Residual norm 3.283104975114e-46
>      1 SNES Function norm 1.867771204856e-44
> 3 TS dt 0.01 time 0.03
>      0 SNES Function norm 1.644541571708e-44
>        0 KSP Residual norm 1.969573456719e-46
>        1 KSP Residual norm 1.297976290541e-61
>      1 SNES Function norm 1.780215821230e-59
> 4 TS dt 0.01 time 0.04
>      0 SNES Function norm 1.700894451833e-59
> 5 TS dt 0.01 time 0.05
>
> ex13.c run:
>
> 0 TS dt 0.01 time 0.
>      0 SNES Function norm 2.327405179696e+02
>        0 KSP Residual norm 9.158498731719e-01
>        1 KSP Residual norm 5.129005976821e-16
>      1 SNES Function norm 1.141297389434e-13
> 1 TS dt 0.01 time 0.01
>      0 SNES Function norm 9.158498731719e+01
>        0 KSP Residual norm 4.156347391374e-01
>        1 KSP Residual norm 1.996081836894e-16
>      1 SNES Function norm 3.609761734594e-14
> 2 TS dt 0.01 time 0.02
>      0 SNES Function norm 4.156347391374e+01
>        0 KSP Residual norm 2.203298795311e-01
>        1 KSP Residual norm 9.047356740098e-17
>      1 SNES Function norm 3.292648977280e-14
> 3 TS dt 0.01 time 0.03
>      0 SNES Function norm 2.203298795311e+01
>        0 KSP Residual norm 1.367458644503e-01
>        1 KSP Residual norm 5.055030661919e-17
>      1 SNES Function norm 1.272304808486e-14
> 4 TS dt 0.01 time 0.04
>      0 SNES Function norm 1.367458644503e+01
>        0 KSP Residual norm 9.695393464180e-02
>        1 KSP Residual norm 1.918401734795e-17
>      1 SNES Function norm 1.130247628824e-14
> 5 TS dt 0.01 time 0.05
>
> I rechecked the Python code against the C one but I did not find the
> source of the issue.
>
> # Example 13 petsc TS
> import sys, petsc4py
> petsc4py.init(sys.argv)
>
> from petsc4py import PETSc
> from mpi4py import MPI
> import numpy as np
> import math
>
> def RHS_func(ts, t, X, xdot, F):
>      da = ts.getDM()
>      localU = da.getLocalVec()
>
>      da.globalToLocal(X, localU)
>
>      mx, my = da.getSizes()
>
>      hx, hy = [1.0/(m-1) for m in [mx, my]]
>      sx = 1.0/(hx*hx)
>      sy = 1.0/(hy*hy)
>
>      uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
>      f = F.getArray(readonly=0).reshape(8, 8, order='C')
>
>      (xs, xm), (ys, ym) = da.getRanges()
>      for j in range(ys, ym):
>          for i in range(xs, xm):
>              if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
>                  f[i,j] = uarray[i,j]
>                  continue
>              u = uarray[i,j]
>              uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
>              uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
>              f[i, j] = uxx + uyy
>      F.assemble()
>      #PETSc.Log.logFlops(11.0*xm*ym)
>
>
> def Jacobian_func(ts, t, x, xdot, a, A, B):
>      mx, my = da.getSizes()
>      hx = 1.0/(mx-1)
>      hy = 1.0/(my-1)
>      sx = 1.0/(hx*hx)
>      sy = 1.0/(hy*hy)
>
>      B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
>      B.zeroEntries()
>
>      (i0, i1), (j0, j1) = da.getRanges()
>      row = PETSc.Mat.Stencil()
>      col = PETSc.Mat.Stencil()
>
>      for i in range(j0, j1):
>          for j in range(i0, i1):
>              row.index = (i,j)
>              row.field = 0
>              if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
>                  B.setValueStencil(row, row, 1.0)
>              else:
>                  for index, value in [
>                      ((i-1, j),   sx),
>                      ((i+1, j),   sx),
>                      ((i,   j-1), sy),
>                      ((i, j+1), sy),
>                      ((i,   j),  -2*sx - 2*sy)]:
>                      col.index = index
>                      col.field = 0
>                      B.setValueStencil(row, col, value)
>
>      B.assemble()
>      #B.view()
>      if A != B: B.assemble()
>
>      return PETSc.Mat.Structure.SAME_NONZERO_PATTERN
>
> def make_initial_solution(da, U, c):
>      mx, my = da.getSizes()
>      hx = 1.0/(mx-1)
>      hy = 1.0/(my-1)
>      (xs, xm), (ys, ym) = da.getRanges()
>
>      u = U.getArray(readonly=0).reshape(8, 8, order='C')
>
>      for j in range(ys, ym):
>          y = j*hy
>          for i in range(xs, xm):
>              x = i*hx
>              r = math.sqrt((x-0.5)**2+(y-0.5)**2)
>              if r < (0.125):
>                  u[i, j] = math.exp(c*r*r*r)
>              else:
>                  u[i, j] = 0.0
>      U.assemble()
>      #U.view()
>
>
> def monitor(ts, i, t, x):
>      xx = x[:].tolist()
>      history.append((i, t, xx))
>
>
> history = []
> nx = 8
> ny = 8
> da = PETSc.DMDA().create([nx, ny], stencil_type=
> PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)
>
> da.setFromOptions()
> da.setUp()
>
> u = da.createGlobalVec()
> f = u.duplicate()
>
> c = -30.0
>
> ts = PETSc.TS().create()
> ts.setDM(da)
> ts.setType(ts.Type.BEULER)
>
> ts.setIFunction(RHS_func, f)
>
> da.setMatType('aij')
> J = da.createMat()
> ts.setIJacobian(Jacobian_func, J, J)
>
> ftime = 1.0
> ts.setMaxTime(ftime)
> ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)
>
> make_initial_solution(da, u, c)
> dt = 0.01
>
> ts.setMonitor(monitor)
> ts.setTimeStep(dt)
> ts.setFromOptions()
> ts.solve(u)
>
> ftime = ts.getSolveTime()
> steps = ts.getStepNumber()
>
> Thanks.
>
> Nicola
>
>
> --
>
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS
> www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere
> Department
> CARS (Cartography and Remote Sensing) Research Group
> http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY
> ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
>
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally
> privileged information, is intended solely for the use of the intended
> addressees. Opinions, conclusions and other information contained in this
> message, that do not relate to the official business of OGS, shall be
> considered as not given or endorsed by it. Every opinion or advice
> contained in this communication is subject to the terms and conditions
> provided by the agreement governing the engagement with such a client. Any
> use, disclosure, copying or distribution of the contents of this
> communication by a not-intended recipient or in violation of the purposes
> of this communication is strictly prohibited and may be unlawful. For Italy
> only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le
> informazioni contenute in questo messaggio sono riservate ed a uso
> esclusivo del destinatario.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/8b16e505/attachment-0001.html>

From hongzhang at anl.gov  Tue Apr  2 09:54:39 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Tue, 2 Apr 2019 14:54:39 +0000
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
References: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
Message-ID: <83CD5287-183D-48B0-BCF0-1CA4AF6BB85A@anl.gov>


Your python implementation of the residual function and the Jacobian function is intended for solving ODEs in the form x' = f(t,x). So you should use ts.setRHSFunction() and ts.setRHSJacobian() instead of the implicit version.

Hong

> On Apr 2, 2019, at 3:18 AM, Nicola Creati via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello, I have run the two codes using: -pc_type lu -ts_monitor -snes_monitor -ksp_monitor -ts_max_steps 5. The codes start to diverge after the first time step:
> Python run :
> 
> 0 TS dt 0.01 time 0.
>     0 SNES Function norm 2.327405179696e+02
>       0 KSP Residual norm 1.939100379240e+00
>       1 KSP Residual norm 1.013760235597e-15
>     1 SNES Function norm 9.538686413612e-14
> 1 TS dt 0.01 time 0.01
>     0 SNES Function norm 8.565210784140e-14
>       0 KSP Residual norm 1.266678408555e-15
>       1 KSP Residual norm 2.663404219322e-31
>     1 SNES Function norm 5.528402779439e-29
> 2 TS dt 0.01 time 0.02
>     0 SNES Function norm 5.829656072531e-29
>       0 KSP Residual norm 3.805606380832e-31
>       1 KSP Residual norm 3.283104975114e-46
>     1 SNES Function norm 1.867771204856e-44
> 3 TS dt 0.01 time 0.03
>     0 SNES Function norm 1.644541571708e-44
>       0 KSP Residual norm 1.969573456719e-46
>       1 KSP Residual norm 1.297976290541e-61
>     1 SNES Function norm 1.780215821230e-59
> 4 TS dt 0.01 time 0.04
>     0 SNES Function norm 1.700894451833e-59
> 5 TS dt 0.01 time 0.05
> 
> ex13.c run:
> 
> 0 TS dt 0.01 time 0.
>     0 SNES Function norm 2.327405179696e+02
>       0 KSP Residual norm 9.158498731719e-01
>       1 KSP Residual norm 5.129005976821e-16
>     1 SNES Function norm 1.141297389434e-13
> 1 TS dt 0.01 time 0.01
>     0 SNES Function norm 9.158498731719e+01
>       0 KSP Residual norm 4.156347391374e-01
>       1 KSP Residual norm 1.996081836894e-16
>     1 SNES Function norm 3.609761734594e-14
> 2 TS dt 0.01 time 0.02
>     0 SNES Function norm 4.156347391374e+01
>       0 KSP Residual norm 2.203298795311e-01
>       1 KSP Residual norm 9.047356740098e-17
>     1 SNES Function norm 3.292648977280e-14
> 3 TS dt 0.01 time 0.03
>     0 SNES Function norm 2.203298795311e+01
>       0 KSP Residual norm 1.367458644503e-01
>       1 KSP Residual norm 5.055030661919e-17
>     1 SNES Function norm 1.272304808486e-14
> 4 TS dt 0.01 time 0.04
>     0 SNES Function norm 1.367458644503e+01
>       0 KSP Residual norm 9.695393464180e-02
>       1 KSP Residual norm 1.918401734795e-17
>     1 SNES Function norm 1.130247628824e-14
> 5 TS dt 0.01 time 0.05
> 
> I rechecked the Python code against the C one but I did not find the source of the issue.
> 
> # Example 13 petsc TS
> import sys, petsc4py
> petsc4py.init(sys.argv)
> 
> from petsc4py import PETSc
> from mpi4py import MPI
> import numpy as np
> import math
> 
> def RHS_func(ts, t, X, xdot, F):
>     da = ts.getDM()
>     localU = da.getLocalVec()
> 
>     da.globalToLocal(X, localU)
> 
>     mx, my = da.getSizes()
> 
>     hx, hy = [1.0/(m-1) for m in [mx, my]]
>     sx = 1.0/(hx*hx)
>     sy = 1.0/(hy*hy)
> 
>     uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
>     f = F.getArray(readonly=0).reshape(8, 8, order='C')
> 
>     (xs, xm), (ys, ym) = da.getRanges()
>     for j in range(ys, ym):
>         for i in range(xs, xm):
>             if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
>                 f[i,j] = uarray[i,j]
>                 continue
>             u = uarray[i,j]
>             uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
>             uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
>             f[i, j] = uxx + uyy
>     F.assemble()
>     #PETSc.Log.logFlops(11.0*xm*ym)
> 
> 
> def Jacobian_func(ts, t, x, xdot, a, A, B):
>     mx, my = da.getSizes()
>     hx = 1.0/(mx-1)
>     hy = 1.0/(my-1)
>     sx = 1.0/(hx*hx)
>     sy = 1.0/(hy*hy)
> 
>     B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
>     B.zeroEntries()
> 
>     (i0, i1), (j0, j1) = da.getRanges()
>     row = PETSc.Mat.Stencil()
>     col = PETSc.Mat.Stencil()
> 
>     for i in range(j0, j1):
>         for j in range(i0, i1):
>             row.index = (i,j)
>             row.field = 0
>             if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
>                 B.setValueStencil(row, row, 1.0)
>             else:
>                 for index, value in [
>                     ((i-1, j),   sx),
>                     ((i+1, j),   sx),
>                     ((i,   j-1), sy),
>                     ((i, j+1), sy),
>                     ((i,   j),  -2*sx - 2*sy)]:
>                     col.index = index
>                     col.field = 0
>                     B.setValueStencil(row, col, value)
> 
>     B.assemble()
>     #B.view()
>     if A != B: B.assemble()
> 
>     return PETSc.Mat.Structure.SAME_NONZERO_PATTERN
> 
> def make_initial_solution(da, U, c):
>     mx, my = da.getSizes()
>     hx = 1.0/(mx-1)
>     hy = 1.0/(my-1)
>     (xs, xm), (ys, ym) = da.getRanges()
> 
>     u = U.getArray(readonly=0).reshape(8, 8, order='C')
> 
>     for j in range(ys, ym):
>         y = j*hy
>         for i in range(xs, xm):
>             x = i*hx
>             r = math.sqrt((x-0.5)**2+(y-0.5)**2)
>             if r < (0.125):
>                 u[i, j] = math.exp(c*r*r*r)
>             else:
>                 u[i, j] = 0.0
>     U.assemble()
>     #U.view()
> 
> 
> def monitor(ts, i, t, x):
>     xx = x[:].tolist()
>     history.append((i, t, xx))
> 
> 
> history = []
> nx = 8
> ny = 8
> da = PETSc.DMDA().create([nx, ny], stencil_type= PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)
> 
> da.setFromOptions()
> da.setUp()
> 
> u = da.createGlobalVec()
> f = u.duplicate()
> 
> c = -30.0
> 
> ts = PETSc.TS().create()
> ts.setDM(da)
> ts.setType(ts.Type.BEULER)
> 
> ts.setIFunction(RHS_func, f)
> 
> da.setMatType('aij')
> J = da.createMat()
> ts.setIJacobian(Jacobian_func, J, J)
> 
> ftime = 1.0
> ts.setMaxTime(ftime)
> ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)
> 
> make_initial_solution(da, u, c)
> dt = 0.01
> 
> ts.setMonitor(monitor)
> ts.setTimeStep(dt)
> ts.setFromOptions()
> ts.solve(u)
> 
> ftime = ts.getSolveTime()
> steps = ts.getStepNumber()
> 
> Thanks.
> 
> Nicola
> 
> 
> -- 
> 
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
> CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
> 
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.
> 


From bsmith at mcs.anl.gov  Tue Apr  2 13:08:34 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 2 Apr 2019 18:08:34 +0000
Subject: [petsc-users] Local and global size of IS
In-Reply-To: <CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>
References: <CA+vu_Xh6Qr8c=mrkDccbxBNgd3KDO8B92S7Z_SRTXFcby+43BQ@mail.gmail.com>
	<202187C0-0CD5-46F6-BE73-4F48243BE119@anl.gov>
	<CA+vu_XgTrnuYp2h514kezrcJLTpZBn7nzSWjV9qsPjGYSWr22Q@mail.gmail.com>
Message-ID: <44B4F279-93FC-4E8D-8791-D310CAF7D1B5@mcs.anl.gov>



> On Apr 2, 2019, at 5:54 AM, Eda Oktay <eda.oktay at metu.edu.tr> wrote:
> 
> Hi Barry,
> 
> I did what you said but now I get the error in ISSetPermutation that index set is not a permutation.

   When you create the IS you need to pass in PETSC_COMM_WORLD as the MPI_Comm, not PETSC_COMM_SELF. Otherwise it thinks the IS is sequential and checks if it is a permutation and hence generates the error.

> ? think it is because since I selected indices on each process as you said. I did the following, did I understand you wrong:
> 
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>  
>   PetscInt *idxx,ss;
>   ss = siz/size;
>   PetscMalloc1(ss,&idxx);
>   if (rank != size-1) {
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
> 
>   } else {
>       
>     j =0;
>     for (i=rank*ss; i<siz; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>       
>   }
> 
> Smith, Barry F. <bsmith at mcs.anl.gov>, 31 Mar 2019 Paz, 01:47 tarihinde ?unu yazd?:
> 
> 
> > On Mar 27, 2019, at 7:33 AM, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello,
> > 
> > I am trying to permute a matrix A(of size 2n*2n) according to a sorted eigenvector vr (of size 2n) in parallel using 2 processors (processor number can change).
> 
>     I don't understand your rational for wanting to permute the matrix rows/columns based on a sorted eigenvector? 
> 
> > However, I get an error in MatPermute line stating that arguments are out of range and a new nonzero caused a malloc even if I used MatSetOption. 
> > 
> > I discovered that this may be because of the unequality of local sizes of is (and newIS) and local size of A. 
> > 
> > Since I allocate index set idx according to size of the vector vr, global size of is becomes 2n and the local size is also 2n (I think it should be n since both A and vr has local sizes n because of the number of processors). If I change the size of idx, then because of VecGetArrayRead, I think is is created wrongly.
> > 
> > So, how can I make both global and local sizes of is,newIS and A?
> > 
> > Below, you can see the part of my program.
> > 
> > Thanks,
> > 
> > Eda
> >  
> >  ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);                            
> >   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
> >   for (i=0; i<siz;i++) idx[i] = i;
> 
>    VecGetArray only gives you access to the local portion of the vector, not the entire vector so you cannot do the next two lines of code
> >   ierr = VecGetArrayRead(vr,&avr);CHKERRQ(ierr);
> 
>    The sort routine is sequential; it knows nothing about parallelism so each process is just sorting its part. Hence this code won't work as expected
> 
> >   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr); 
> 
>    If you need to sort the parallel vector and get the entire permutation then you need to do the following 
> 
> 1)   communicate the entire vector to each process, you can use VecScatterCreateToAll() for this
> 2)   call VecGetArray on the new vector (that contains all entries on each process)
> 3)   Call PetscSortRealWithPermutation() on the entire vector on each process
> 4)   select out a piece of the resulting indices idx on each process; for example with two processes I think rank = 0 would get the first half of the idx and rank = 1 would get the second half.
> >                  
> >   ierr = ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);  
> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
> 
>   You don't need to duplicate the is, just pass it in twice.
> 
> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
> 
>    You should definitely not need the next line. The A matrix is untouched by the subroutine call so you don't need to change its allocation options
> 
> > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);  
> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);  
> 


From bsmith at mcs.anl.gov  Tue Apr  2 13:29:58 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 2 Apr 2019 18:29:58 +0000
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <CAMYG4Gk9V7oCrRopNo=UMyZ+acJ7hqdx8-L05=Tms=Ce1q_Cpg@mail.gmail.com>
References: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
	<CAMYG4Gk9V7oCrRopNo=UMyZ+acJ7hqdx8-L05=Tms=Ce1q_Cpg@mail.gmail.com>
Message-ID: <0866A9FF-B5F6-496D-AC92-516A998D1DE0@anl.gov>


  More explicitly. In the C function you are copying from it uses 

ierr = TSSetRHSFunction(ts,r,RHSFunction,&user);CHKERRQ(ierr);
....
ierr = TSSetRHSJacobian(ts,J,J,RHSJacobian,NULL);CHKERRQ(ierr);

  But in your code you use

ts.setIFunction(RHS_func, f)
ts.setIJacobian(Jacobian_func, J, J)


  This simply will not work, the SetRHS* and SetI* functions are not interchangeable. You functions you provide in the two cases have to be different. See the manual pages for the various routines. And the PETSc users manual.

First get the python working with SetRHS* (confirm identical results to the C program) and then if you want to use the SetI* you must rewrite the RHS_func and Jacobian_func functions that you provide following the explanation in the manual pages and user manual.

   Barry



> On Apr 2, 2019, at 9:07 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Tue, Apr 2, 2019 at 4:19 AM Nicola Creati via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello, I have run the two codes using: -pc_type lu -ts_monitor 
> -snes_monitor -ksp_monitor -ts_max_steps 5. The codes start to diverge 
> after the first time step:
> 
> In RHS_Function, you never use xdot, but this is an implicit TS. Thus, you are solving the Laplace
> equation, not the heat equation, and you get the exact answer after one step.
> 
>   Matt
>  
> Python run :
> 
> 0 TS dt 0.01 time 0.
>      0 SNES Function norm 2.327405179696e+02
>        0 KSP Residual norm 1.939100379240e+00
>        1 KSP Residual norm 1.013760235597e-15
>      1 SNES Function norm 9.538686413612e-14
> 1 TS dt 0.01 time 0.01
>      0 SNES Function norm 8.565210784140e-14
>        0 KSP Residual norm 1.266678408555e-15
>        1 KSP Residual norm 2.663404219322e-31
>      1 SNES Function norm 5.528402779439e-29
> 2 TS dt 0.01 time 0.02
>      0 SNES Function norm 5.829656072531e-29
>        0 KSP Residual norm 3.805606380832e-31
>        1 KSP Residual norm 3.283104975114e-46
>      1 SNES Function norm 1.867771204856e-44
> 3 TS dt 0.01 time 0.03
>      0 SNES Function norm 1.644541571708e-44
>        0 KSP Residual norm 1.969573456719e-46
>        1 KSP Residual norm 1.297976290541e-61
>      1 SNES Function norm 1.780215821230e-59
> 4 TS dt 0.01 time 0.04
>      0 SNES Function norm 1.700894451833e-59
> 5 TS dt 0.01 time 0.05
> 
> ex13.c run:
> 
> 0 TS dt 0.01 time 0.
>      0 SNES Function norm 2.327405179696e+02
>        0 KSP Residual norm 9.158498731719e-01
>        1 KSP Residual norm 5.129005976821e-16
>      1 SNES Function norm 1.141297389434e-13
> 1 TS dt 0.01 time 0.01
>      0 SNES Function norm 9.158498731719e+01
>        0 KSP Residual norm 4.156347391374e-01
>        1 KSP Residual norm 1.996081836894e-16
>      1 SNES Function norm 3.609761734594e-14
> 2 TS dt 0.01 time 0.02
>      0 SNES Function norm 4.156347391374e+01
>        0 KSP Residual norm 2.203298795311e-01
>        1 KSP Residual norm 9.047356740098e-17
>      1 SNES Function norm 3.292648977280e-14
> 3 TS dt 0.01 time 0.03
>      0 SNES Function norm 2.203298795311e+01
>        0 KSP Residual norm 1.367458644503e-01
>        1 KSP Residual norm 5.055030661919e-17
>      1 SNES Function norm 1.272304808486e-14
> 4 TS dt 0.01 time 0.04
>      0 SNES Function norm 1.367458644503e+01
>        0 KSP Residual norm 9.695393464180e-02
>        1 KSP Residual norm 1.918401734795e-17
>      1 SNES Function norm 1.130247628824e-14
> 5 TS dt 0.01 time 0.05
> 
> I rechecked the Python code against the C one but I did not find the 
> source of the issue.
> 
> # Example 13 petsc TS
> import sys, petsc4py
> petsc4py.init(sys.argv)
> 
> from petsc4py import PETSc
> from mpi4py import MPI
> import numpy as np
> import math
> 
> def RHS_func(ts, t, X, xdot, F):
>      da = ts.getDM()
>      localU = da.getLocalVec()
> 
>      da.globalToLocal(X, localU)
> 
>      mx, my = da.getSizes()
> 
>      hx, hy = [1.0/(m-1) for m in [mx, my]]
>      sx = 1.0/(hx*hx)
>      sy = 1.0/(hy*hy)
> 
>      uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
>      f = F.getArray(readonly=0).reshape(8, 8, order='C')
> 
>      (xs, xm), (ys, ym) = da.getRanges()
>      for j in range(ys, ym):
>          for i in range(xs, xm):
>              if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
>                  f[i,j] = uarray[i,j]
>                  continue
>              u = uarray[i,j]
>              uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
>              uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
>              f[i, j] = uxx + uyy
>      F.assemble()
>      #PETSc.Log.logFlops(11.0*xm*ym)
> 
> 
> def Jacobian_func(ts, t, x, xdot, a, A, B):
>      mx, my = da.getSizes()
>      hx = 1.0/(mx-1)
>      hy = 1.0/(my-1)
>      sx = 1.0/(hx*hx)
>      sy = 1.0/(hy*hy)
> 
>      B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
>      B.zeroEntries()
> 
>      (i0, i1), (j0, j1) = da.getRanges()
>      row = PETSc.Mat.Stencil()
>      col = PETSc.Mat.Stencil()
> 
>      for i in range(j0, j1):
>          for j in range(i0, i1):
>              row.index = (i,j)
>              row.field = 0
>              if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
>                  B.setValueStencil(row, row, 1.0)
>              else:
>                  for index, value in [
>                      ((i-1, j),   sx),
>                      ((i+1, j),   sx),
>                      ((i,   j-1), sy),
>                      ((i, j+1), sy),
>                      ((i,   j),  -2*sx - 2*sy)]:
>                      col.index = index
>                      col.field = 0
>                      B.setValueStencil(row, col, value)
> 
>      B.assemble()
>      #B.view()
>      if A != B: B.assemble()
> 
>      return PETSc.Mat.Structure.SAME_NONZERO_PATTERN
> 
> def make_initial_solution(da, U, c):
>      mx, my = da.getSizes()
>      hx = 1.0/(mx-1)
>      hy = 1.0/(my-1)
>      (xs, xm), (ys, ym) = da.getRanges()
> 
>      u = U.getArray(readonly=0).reshape(8, 8, order='C')
> 
>      for j in range(ys, ym):
>          y = j*hy
>          for i in range(xs, xm):
>              x = i*hx
>              r = math.sqrt((x-0.5)**2+(y-0.5)**2)
>              if r < (0.125):
>                  u[i, j] = math.exp(c*r*r*r)
>              else:
>                  u[i, j] = 0.0
>      U.assemble()
>      #U.view()
> 
> 
> def monitor(ts, i, t, x):
>      xx = x[:].tolist()
>      history.append((i, t, xx))
> 
> 
> history = []
> nx = 8
> ny = 8
> da = PETSc.DMDA().create([nx, ny], stencil_type= 
> PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)
> 
> da.setFromOptions()
> da.setUp()
> 
> u = da.createGlobalVec()
> f = u.duplicate()
> 
> c = -30.0
> 
> ts = PETSc.TS().create()
> ts.setDM(da)
> ts.setType(ts.Type.BEULER)
> 
> ts.setIFunction(RHS_func, f)
> 
> da.setMatType('aij')
> J = da.createMat()
> ts.setIJacobian(Jacobian_func, J, J)
> 
> ftime = 1.0
> ts.setMaxTime(ftime)
> ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)
> 
> make_initial_solution(da, u, c)
> dt = 0.01
> 
> ts.setMonitor(monitor)
> ts.setTimeStep(dt)
> ts.setFromOptions()
> ts.solve(u)
> 
> ftime = ts.getSolveTime()
> steps = ts.getStepNumber()
> 
> Thanks.
> 
> Nicola
> 
> 
> -- 
> 
> Nicola Creati
> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
> CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
> off.   +39 040 2140 213
> fax.   +39 040 327307
> 
> _____________________________________________________________________
> This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From Fabian.Jakub at physik.uni-muenchen.de  Tue Apr  2 15:49:40 2019
From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub)
Date: Tue, 2 Apr 2019 22:49:40 +0200
Subject: [petsc-users] Fwd: Error in HDF5 dumps of DMPlex labels
In-Reply-To: <c748a0c9-74c2-4cdb-3f1a-81003542d70f@physik.uni-muenchen.de>
References: <c748a0c9-74c2-4cdb-3f1a-81003542d70f@physik.uni-muenchen.de>
Message-ID: <699b94b6-0ef3-ae6b-ff4c-2fcb36c80fc3@physik.uni-muenchen.de>

Hi there,

I'm not sure if this message got lost (have sent it to petsc-dev, trying
users list now) or if I just did not get the replies.
Anyway, the issue that is attached below is unresolved in the current
master branch and I am curious about your input.

Many Thanks,

Fabian


-------- Weitergeleitete Nachricht --------
Betreff: [petsc-dev] Error in HDF5 dumps of DMPlex labels
Datum: Thu, 14 Feb 2019 14:53:22 +0100
Von: Fabian.Jakub via petsc-dev <petsc-dev at mcs.anl.gov>
Antwort an: Fabian.Jakub <Fabian.Jakub at physik.uni-muenchen.de>
An: petsc-dev <petsc-dev at mcs.anl.gov>

Dear Petsc Team!

I had an issue when writing out DMPlex objects through hdf5.


This comes from a DMLabel that has only entries on non-local mesh points.
The DMLabel write only includes local parts of the label and so leads to
a zero sized write for the index set.
This seems to be fine except that the hdf5 chunksize is set to zero
which is not allowed.

I added a minimal example to illustrate the error.
It creates a 2D DMPlex in serial, distributes it, labels the nonlocal
points in the mesh and dumps it via PetscObjectViewer to HDF5.
Run with:

   make plex.h5

I also attached a quick fix to override the chunksize.

Please let me know if you anything extra and also if this is expected
behavior... I could certainly with the fact that DMLabel is not supposed
to work this way.

Many thanks,

Fabian


-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-fix-hdf5-chunksizes-of-0.patch
Type: text/x-patch
Size: 1036 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/cb71579b/attachment-0002.bin>
-------------- next part --------------
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

plex.xmf:: plex.h5
		${PETSC_DIR}/lib/petsc/bin/petsc_gen_xdmf.py plex.h5

plex.h5:: hdf5_issue_nonlocal_only_labels
		mpirun -np 2 ./hdf5_issue_nonlocal_only_labels -show_labeled_plex ::ascii_info_detail
		mpirun -np 2 ./hdf5_issue_nonlocal_only_labels -show_labeled_plex hdf5:plex.h5

hdf5_issue_nonlocal_only_labels:: hdf5_issue_nonlocal_only_labels.F90
		${PETSC_FCOMPILE} -c hdf5_issue_nonlocal_only_labels.F90
		${FLINKER} hdf5_issue_nonlocal_only_labels.o -o hdf5_issue_nonlocal_only_labels ${PETSC_LIB}

clean::
		rm -rf *.o plex.h5 plex.xmf hdf5_issue_nonlocal_only_labels
-------------- next part --------------
HDF5-DIAG: Error detected in HDF5 (1.8.18) MPI-process 0:
#000: H5Pdcpl.c line 870 in H5Pset_chunk(): all chunk dimensions must be positive
major: Invalid arguments to routine
minor: Out of range
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error in HDF5 call H5Pset_chunk() Status -1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1549-g9b78045  GIT Date: 2019-02-13 11:10:12 -0600
[0]PETSC ERROR: plex_prism on a debug_gcc named met-ws-970r17 by Fabian.Jakub Thu Feb 14 14:39:13 2019
[0]PETSC ERROR: Configure options --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 --with-cxx=/usr/bin/mpicxx --with-fortran --with-fortran-interfaces --with-shared-libraries=1 --download-hdf5 --download-netcdf=netcdf-c-4.6.2.tar.gz --download-zlib --download-szlib --download-openblas --download-metis --download-parmetis --download-ptscotch --no-download-petsc4py --download-ml --download-hypre --with-debugging=1 COPTFLAGS="-O2 -g" FOPTFLAGS="-O2 -g"
[0]PETSC ERROR: #1 ISView_General_HDF5() line 288 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #2 ISView_General() line 513 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/impls/general/general.c
[0]PETSC ERROR: #3 ISView() line 1015 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/interface/index.c
[0]PETSC ERROR: #4 DMPlexWriteLabels_HDF5_Static() line 668 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plexhdf5.c
[0]PETSC ERROR: #5 DMPlexView_HDF5_Internal() line 692 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plexhdf5.c
[0]PETSC ERROR: #6 DMView_Plex() line 1103 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plex.c
[0]PETSC ERROR: #7 DMView() line 904 in /software/meteo/xenial/x86_64/petsc/master/src/dm/interface/dm.c
[0]PETSC ERROR: #8 PetscObjectView() line 100 in /software/meteo/xenial/x86_64/petsc/master/src/sys/objects/destroy.c
[0]PETSC ERROR: #9 PetscObjectViewFromOptions() line 136 in /software/meteo/xenial/x86_64/petsc/master/src/sys/objects/destroy.c
[0]PETSC ERROR: #10 User provided function() line 0 in User file


HDF5-DIAG: Error detected in HDF5 (1.8.18) MPI-process 1:
#000: H5Pdcpl.c line 870 in H5Pset_chunk(): all chunk dimensions must be positive
major: Invalid arguments to routine
minor: Out of range
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[1]PETSC ERROR: Error in HDF5 call H5Pset_chunk() Status -1
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1549-g9b78045  GIT Date: 2019-02-13 11:10:12 -0600
[1]PETSC ERROR: plex_prism on a debug_gcc named met-ws-970r17 by Fabian.Jakub Thu Feb 14 14:39:13 2019
[1]PETSC ERROR: Configure options --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 --with-cxx=/usr/bin/mpicxx --with-fortran --with-fortran-interfaces --with-shared-libraries=1 --download-hdf5 --download-netcdf=netcdf-c-4.6.2.tar.gz --download-zlib --download-szlib --download-openblas --download-metis --download-parmetis --download-ptscotch --no-download-petsc4py --download-ml --download-hypre --with-debugging=1 COPTFLAGS="-O2 -g" FOPTFLAGS="-O2 -g"
[1]PETSC ERROR: #1 ISView_General_HDF5() line 288 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: #2 ISView_General() line 513 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/impls/general/general.c
[1]PETSC ERROR: #3 ISView() line 1015 in /software/meteo/xenial/x86_64/petsc/master/src/vec/is/is/interface/index.c
[1]PETSC ERROR: #4 DMPlexWriteLabels_HDF5_Static() line 668 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plexhdf5.c
[1]PETSC ERROR: #5 DMPlexView_HDF5_Internal() line 692 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plexhdf5.c
[1]PETSC ERROR: #6 DMView_Plex() line 1103 in /software/meteo/xenial/x86_64/petsc/master/src/dm/impls/plex/plex.c
[1]PETSC ERROR: #7 DMView() line 904 in /software/meteo/xenial/x86_64/petsc/master/src/dm/interface/dm.c
[1]PETSC ERROR: #8 PetscObjectView() line 100 in /software/meteo/xenial/x86_64/petsc/master/src/sys/objects/destroy.c
[1]PETSC ERROR: #9 PetscObjectViewFromOptions() line 136 in /software/meteo/xenial/x86_64/petsc/master/src/sys/objects/destroy.c
[1]PETSC ERROR: #10 User provided function() line 0 in User file
--------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hdf5_issue_nonlocal_only_labels.F90
Type: text/x-fortran
Size: 8502 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/cb71579b/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/cb71579b/attachment-0001.sig>

From bsmith at mcs.anl.gov  Tue Apr  2 16:58:50 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 2 Apr 2019 21:58:50 +0000
Subject: [petsc-users] Fwd: Error in HDF5 dumps of DMPlex labels
In-Reply-To: <699b94b6-0ef3-ae6b-ff4c-2fcb36c80fc3@physik.uni-muenchen.de>
References: <c748a0c9-74c2-4cdb-3f1a-81003542d70f@physik.uni-muenchen.de>
	<699b94b6-0ef3-ae6b-ff4c-2fcb36c80fc3@physik.uni-muenchen.de>
Message-ID: <28193752-F0BD-4786-9414-85494FC327F4@anl.gov>


   Fabian,

     I apologize, your email never got properly processed by someone when it arrived. 

     Thank you for the bug report and in particular for providing a patch. I have submitted a pull request with your patch https://bitbucket.org/petsc/petsc/pull-requests/1494/isview_general_hdf5-crashes-with-a-zero/diff and listed Matt Knepley as a reviewer. The patch looks good to me but I don't understand the exactly subtleties of HDF chunk sizes. Hopefully the patch is acceptable to Matt and he will review it, otherwise he will have to propose an alternative fix. 

     After approval and compile testing it will be put in the maint and master branches and be in the 3.11.1 patch release of PETSc.

   Barry


> On Apr 2, 2019, at 3:49 PM, Fabian.Jakub via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi there,
> 
> I'm not sure if this message got lost (have sent it to petsc-dev, trying
> users list now) or if I just did not get the replies.
> Anyway, the issue that is attached below is unresolved in the current
> master branch and I am curious about your input.
> 
> Many Thanks,
> 
> Fabian
> 
> 
> -------- Weitergeleitete Nachricht --------
> Betreff: [petsc-dev] Error in HDF5 dumps of DMPlex labels
> Datum: Thu, 14 Feb 2019 14:53:22 +0100
> Von: Fabian.Jakub via petsc-dev <petsc-dev at mcs.anl.gov>
> Antwort an: Fabian.Jakub <Fabian.Jakub at physik.uni-muenchen.de>
> An: petsc-dev <petsc-dev at mcs.anl.gov>
> 
> Dear Petsc Team!
> 
> I had an issue when writing out DMPlex objects through hdf5.
> 
> 
> This comes from a DMLabel that has only entries on non-local mesh points.
> The DMLabel write only includes local parts of the label and so leads to
> a zero sized write for the index set.
> This seems to be fine except that the hdf5 chunksize is set to zero
> which is not allowed.
> 
> I added a minimal example to illustrate the error.
> It creates a 2D DMPlex in serial, distributes it, labels the nonlocal
> points in the mesh and dumps it via PetscObjectViewer to HDF5.
> Run with:
> 
>   make plex.h5
> 
> I also attached a quick fix to override the chunksize.
> 
> Please let me know if you anything extra and also if this is expected
> behavior... I could certainly with the fact that DMLabel is not supposed
> to work this way.
> 
> Many thanks,
> 
> Fabian
> 
> 
> <0001-fix-hdf5-chunksizes-of-0.patch><Makefile.txt><error_msg.txt><hdf5_issue_nonlocal_only_labels.F90>


From fdkong.jd at gmail.com  Tue Apr  2 22:59:46 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 2 Apr 2019 21:59:46 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
Message-ID: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>

Hi All,

There were some error messages when using PetscSFReduceBegin with MPI_CHAR.

ierr =
PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);


My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR?
If not, should we document somewhere?

Thanks

Fande,


[*0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------*
*[0]PETSC ERROR: No support for this operation for this object type*
*[0]PETSC ERROR: No support for type size not divisible by 4*
*[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
<http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
*[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
 GIT Date: 2019-04-02 17:37:18 -0600*
*[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------*
*[1]PETSC ERROR: No support for this operation for this object type*
*[1]PETSC ERROR: No support for type size not divisible by 4*
*[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
<http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
*[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
 GIT Date: 2019-04-02 17:37:18 -0600*
*[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once
named fn605731.local by kongf Tue Apr  2 21:48:41 2019*
*[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
--with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
--download-parmetis=1 --download-superlu_dist=1
PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
--download-party --download-chaco --with-cxx-dialect=C++11*
*[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named
fn605731.local by kongf Tue Apr  2 21:48:41 2019*
*[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
--with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
--download-parmetis=1 --download-superlu_dist=1
PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
--download-party --download-chaco --with-cxx-dialect=C++11*
*[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
/Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
*[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
/Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
*[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
/Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
*[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
/Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
*[0]PETSC ERROR: #7 MatPtAP() line 9429 in
/Users/kongf/projects/petsc/src/mat/interface/matrix.c*
*[0]PETSC ERROR: #8 main() line 58 in
/Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
*[0]PETSC ERROR: PETSc Option Table entries:*
*[0]PETSC ERROR: -matptap_via allatonce*
*[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------*
*[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
/Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
*[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
/Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
*[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
/Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
*[1]PETSC ERROR: #7 MatPtAP() line 9429 in
/Users/kongf/projects/petsc/src/mat/interface/matrix.c*
*[1]PETSC ERROR: #8 main() line 58 in
/Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
*[1]PETSC ERROR: PETSc Option Table entries:*
*[1]PETSC ERROR: -matptap_via allatonce*
*[1]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------*
*--------------------------------------------------------------------------*
*MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD*
*with errorcode 56.*

*NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.*
*You may or may not see output from other processes, depending on*
*exactly when Open MPI kills them.*
*--------------------------------------------------------------------------*
*[fn605731.local:78133] 1 more process has sent help message
help-mpi-api.txt / mpi-abort*
*[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0
to see all help / error messages*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190402/f93dfbc9/attachment.html>

From jed at jedbrown.org  Tue Apr  2 23:06:48 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 02 Apr 2019 22:06:48 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
Message-ID: <87y34rhgvb.fsf@jedbrown.org>

We can add it easily.  Would it be enough to add it to petsc-3.11.*?
(I'd rather not backport to an earlier version, for which we presumably
won't have any more maintenance releases.)

Fande Kong via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi All,
>
> There were some error messages when using PetscSFReduceBegin with MPI_CHAR.
>
> ierr =
> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>
>
> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR?
> If not, should we document somewhere?
>
> Thanks
>
> Fande,
>
>
> [*0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> *[0]PETSC ERROR: No support for this operation for this object type*
> *[0]PETSC ERROR: No support for type size not divisible by 4*
> *[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
> *[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>  GIT Date: 2019-04-02 17:37:18 -0600*
> *[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> *[1]PETSC ERROR: No support for this operation for this object type*
> *[1]PETSC ERROR: No support for type size not divisible by 4*
> *[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
> *[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>  GIT Date: 2019-04-02 17:37:18 -0600*
> *[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once
> named fn605731.local by kongf Tue Apr  2 21:48:41 2019*
> *[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
> --download-parmetis=1 --download-superlu_dist=1
> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
> --download-party --download-chaco --with-cxx-dialect=C++11*
> *[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named
> fn605731.local by kongf Tue Apr  2 21:48:41 2019*
> *[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
> --download-parmetis=1 --download-superlu_dist=1
> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
> --download-party --download-chaco --with-cxx-dialect=C++11*
> *[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
> *[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[0]PETSC ERROR: #7 MatPtAP() line 9429 in
> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
> *[0]PETSC ERROR: #8 main() line 58 in
> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
> *[0]PETSC ERROR: PETSc Option Table entries:*
> *[0]PETSC ERROR: -matptap_via allatonce*
> *[0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------*
> *[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
> *[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[1]PETSC ERROR: #7 MatPtAP() line 9429 in
> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
> *[1]PETSC ERROR: #8 main() line 58 in
> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
> *[1]PETSC ERROR: PETSc Option Table entries:*
> *[1]PETSC ERROR: -matptap_via allatonce*
> *[1]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------*
> *--------------------------------------------------------------------------*
> *MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD*
> *with errorcode 56.*
>
> *NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.*
> *You may or may not see output from other processes, depending on*
> *exactly when Open MPI kills them.*
> *--------------------------------------------------------------------------*
> *[fn605731.local:78133] 1 more process has sent help message
> help-mpi-api.txt / mpi-abort*
> *[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0
> to see all help / error messages*

From fdkong.jd at gmail.com  Tue Apr  2 23:39:48 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 2 Apr 2019 22:39:48 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <87y34rhgvb.fsf@jedbrown.org>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<87y34rhgvb.fsf@jedbrown.org>
Message-ID: <FC2405C7-A454-4168-A529-1320823FA3DE@gmail.com>

I am working on petsc master. So it should be fine to have it in 3.11

Fande 


> On Apr 2, 2019, at 10:06 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> We can add it easily.  Would it be enough to add it to petsc-3.11.*?
> (I'd rather not backport to an earlier version, for which we presumably
> won't have any more maintenance releases.)
> 
> Fande Kong via petsc-users <petsc-users at mcs.anl.gov> writes:
> 
>> Hi All,
>> 
>> There were some error messages when using PetscSFReduceBegin with MPI_CHAR.
>> 
>> ierr =
>> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>> 
>> 
>> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR?
>> If not, should we document somewhere?
>> 
>> Thanks
>> 
>> Fande,
>> 
>> 
>> [*0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------*
>> *[0]PETSC ERROR: No support for this operation for this object type*
>> *[0]PETSC ERROR: No support for type size not divisible by 4*
>> *[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
>> *[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>> GIT Date: 2019-04-02 17:37:18 -0600*
>> *[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------*
>> *[1]PETSC ERROR: No support for this operation for this object type*
>> *[1]PETSC ERROR: No support for type size not divisible by 4*
>> *[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
>> *[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>> GIT Date: 2019-04-02 17:37:18 -0600*
>> *[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once
>> named fn605731.local by kongf Tue Apr  2 21:48:41 2019*
>> *[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
>> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
>> --download-parmetis=1 --download-superlu_dist=1
>> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
>> --download-party --download-chaco --with-cxx-dialect=C++11*
>> *[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named
>> fn605731.local by kongf Tue Apr  2 21:48:41 2019*
>> *[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
>> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
>> --download-parmetis=1 --download-superlu_dist=1
>> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
>> --download-party --download-chaco --with-cxx-dialect=C++11*
>> *[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>> *[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
>> *[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>> *[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>> *[0]PETSC ERROR: #7 MatPtAP() line 9429 in
>> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
>> *[0]PETSC ERROR: #8 main() line 58 in
>> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
>> *[0]PETSC ERROR: PETSc Option Table entries:*
>> *[0]PETSC ERROR: -matptap_via allatonce*
>> *[0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------*
>> *[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
>> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
>> *[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>> *[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>> *[1]PETSC ERROR: #7 MatPtAP() line 9429 in
>> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
>> *[1]PETSC ERROR: #8 main() line 58 in
>> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
>> *[1]PETSC ERROR: PETSc Option Table entries:*
>> *[1]PETSC ERROR: -matptap_via allatonce*
>> *[1]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------*
>> *--------------------------------------------------------------------------*
>> *MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD*
>> *with errorcode 56.*
>> 
>> *NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.*
>> *You may or may not see output from other processes, depending on*
>> *exactly when Open MPI kills them.*
>> *--------------------------------------------------------------------------*
>> *[fn605731.local:78133] 1 more process has sent help message
>> help-mpi-api.txt / mpi-abort*
>> *[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0
>> to see all help / error messages*

From jed at jedbrown.org  Tue Apr  2 23:47:49 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 02 Apr 2019 22:47:49 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <FC2405C7-A454-4168-A529-1320823FA3DE@gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<87y34rhgvb.fsf@jedbrown.org>
	<FC2405C7-A454-4168-A529-1320823FA3DE@gmail.com>
Message-ID: <87sguzheyy.fsf@jedbrown.org>

Fande Kong <fdkong.jd at gmail.com> writes:

> I am working on petsc master. So it should be fine to have it in 3.11

Cool, I'd rather just do it in 'master'.

From fdkong.jd at gmail.com  Tue Apr  2 23:52:04 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 2 Apr 2019 22:52:04 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <87sguzheyy.fsf@jedbrown.org>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<87y34rhgvb.fsf@jedbrown.org>
	<FC2405C7-A454-4168-A529-1320823FA3DE@gmail.com>
	<87sguzheyy.fsf@jedbrown.org>
Message-ID: <714DD61D-C0D9-4D61-BB69-22609C51D7CD@gmail.com>

Thanks, Jed,

Please let me know when the patch is in master.

Fande 


> On Apr 2, 2019, at 10:47 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Fande Kong <fdkong.jd at gmail.com> writes:
> 
>> I am working on petsc master. So it should be fine to have it in 3.11
> 
> Cool, I'd rather just do it in 'master'.

From jed at jedbrown.org  Wed Apr  3 00:26:32 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 02 Apr 2019 23:26:32 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <714DD61D-C0D9-4D61-BB69-22609C51D7CD@gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<87y34rhgvb.fsf@jedbrown.org>
	<FC2405C7-A454-4168-A529-1320823FA3DE@gmail.com>
	<87sguzheyy.fsf@jedbrown.org>
	<714DD61D-C0D9-4D61-BB69-22609C51D7CD@gmail.com>
Message-ID: <87pnq3hd6f.fsf@jedbrown.org>

You can try branch 'jed/feature-sf-char'; not tested yet.

Fande Kong <fdkong.jd at gmail.com> writes:

> Thanks, Jed,
>
> Please let me know when the patch is in master.
>
> Fande 
>
>
>> On Apr 2, 2019, at 10:47 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>> Fande Kong <fdkong.jd at gmail.com> writes:
>> 
>>> I am working on petsc master. So it should be fine to have it in 3.11
>> 
>> Cool, I'd rather just do it in 'master'.

From zonexo at gmail.com  Wed Apr  3 00:48:11 2019
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 3 Apr 2019 13:48:11 +0800
Subject: [petsc-users] Unable to read in values thru namelist in Fortran
 after using PETSc 64bit in linux
In-Reply-To: <686A13FC-7B4E-4F7C-84E0-3A0FCFC08FA1@mcs.anl.gov>
References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com>
	<0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov>
	<22565566-590d-7c89-390a-99557ce05382@gmail.com>
	<alpine.LFD.2.21.1809180034010.2752@asterix>
	<cdde09d2-f3eb-21c9-d0a7-fda195c17be5@gmail.com>
	<C4D68689-DDF9-4C94-9C20-FB10F2C748FD@mcs.anl.gov>
	<69d01b45-050f-2729-3015-14f3d80202d5@gmail.com>
	<686A13FC-7B4E-4F7C-84E0-3A0FCFC08FA1@mcs.anl.gov>
Message-ID: <1c3ff9e7-ebf8-a0e6-c2e9-5d2307bdf656@gmail.com>

Hi,

I just encounter a mpi_allgatherv problem when using 64bit PETSc:

call 
MPI_ALLGATHERV(tmp_mpi_data,counter,MPIU_INTEGER,tmp_mpi_data2,counter_global,idisp,MPIU_INTEGER,MPI_COMM_WORLD,ierr)

The error is:

Fatal error in PMPI_Allgatherv: Message truncated, error stack:
PMPI_Allgatherv(1452).....: MPI_Allgatherv(sbuf=0x75cfdc0, scount=1119, 
dtype=0x4c000831, rbuf=0x75d8600, rcounts=0x7ffffffdb880, 
displs=0x7ffffffdb860, dtype=0x4c000831, MPI_COMM_WORLD) failed
MPIR_Allgatherv_impl(1013): fail failed
MPIR_Allgatherv(967)......: fail failed

MPIR_Allgatherv_intra(222): fail failed
MPIR_Localcopy(107).......: Message truncated; 8952 bytes received but 
buffer size is 8864

The variables are all defined as PetscInt. The strange thing is that I 
did 2-3 MPI_ALLGATHERV which are exactly the same, but only the last one 
got problem. Should I change all these variables to PetscMPIInt?

Also, can I change all PetscInt in the code to PetscMPIInt, except if 
it's labeled void *?


Thank you very much.

Yours sincerely,

================================================
TAY Wee-Beng (Zheng Weiming) ???
Personal research webpage: http://tayweebeng.wixsite.com/website
Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
linkedin: www.linkedin.com/in/tay-weebeng
================================================

On 21/9/2018 2:57 AM, Smith, Barry F. wrote:
>     Yes, you need to go through your code and check each MPI call and make sure you use PetscMPIInt for integer arguments and PetscInt for the void* arguments and also make sure that the data type you use in the MPI calls (when communicating PetscInt) is MPIU_INT.
>
>      You should not need a fancy debugger to find out the crash point. Just a basic debugger like gdb, lldb, or dbx will
>
>     Barry
>
>
>> On Sep 20, 2018, at 8:57 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>> Hi,
>>
>> Sorry I'm still a bit confused. My 64bit code still doesn't work once I use more than 1 procs. It just aborts at some point. I'm been trying to use ARM Forge mpi debugging tool to find the error but it's a bit difficult to back trace.
>>
>> So I should carefully inspect each mpi subroutine or function, is that correct?
>>
>> If it's INT, then I should use PetscMPIInt. If it's labeled void *, I should use PetscInt. Is that so?
>>
>> Thank you very much
>>
>> Yours sincerely,
>>
>> ================================================
>> TAY Wee-Beng ??? (Zheng Weiming)
>> Personal research webpage: http://tayweebeng.wixsite.com/website
>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>> linkedin: www.linkedin.com/in/tay-weebeng
>> ================================================
>>
>> On 19/9/2018 12:49 AM, Smith, Barry F. wrote:
>>>     PetscMPIInt  (or integer) are for all lengths passed to MPI functions; simple look at the prototypes for the MPI function you care about and it will tell you which arguments are integer.  The DATA you are passing into the MPI arrays (which are labeled void * in the manual pages) should be PetscInt.
>>>
>>>
>>>
>>>> On Sep 18, 2018, at 1:39 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER?
>>>>
>>>> In other words, must I assign local_array_length etc as PetscMPIInt?
>>>>
>>>> call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr)
>>>>
>>>> Or is it ok to change all integers from PetscInt to PetscMPIInt?
>>>>
>>>> With the exception of ierr - PetscErrorCode
>>>>
>>>>
>>>> Thank you very much.
>>>>
>>>> Yours sincerely,
>>>>
>>>> ================================================
>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>> ================================================
>>>>
>>>> On 18/9/2018 1:39 PM, Balay, Satish wrote:
>>>>> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html
>>>>>
>>>>> int MPI_Comm_size( MPI_Comm comm, int *size )
>>>>>
>>>>> i.e there is no PetscInt here. [MPI does not know about PETSc datatypes]
>>>>>
>>>>> For convinence we provide PetscMPIInt to keep track of such variables
>>>>> [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F
>>>>>
>>>>> Satish
>>>>>
>>>>> On Tue, 18 Sep 2018, TAY wee-beng wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I managed to find the error appearing after using PETSc 64bit in linux -
>>>>>>
>>>>>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr)
>>>>>>
>>>>>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs)
>>>>>>
>>>>>> Assigning num_procs as integer as the problem.
>>>>>>
>>>>>> Is this supposed to be the case? Or is it a bug?
>>>>>>
>>>>>> Thank you very much.
>>>>>>
>>>>>> Yours sincerely,
>>>>>>
>>>>>> ================================================
>>>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>>> Youtube research showcase:
>>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>>> ================================================
>>>>>>
>>>>>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote:
>>>>>>>         You can try valgrind
>>>>>>>         http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>
>>>>>>>     Barry
>>>>>>>
>>>>>>>
>>>>>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I found that I am unable to read in values thru namelist in Fortran after
>>>>>>>> using PETSc 64bit in linux.
>>>>>>>>
>>>>>>>> I have a parameter txt file which is read in using namelist in Fortran?
>>>>>>>>
>>>>>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config
>>>>>>>> ...
>>>>>>>>
>>>>>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat =
>>>>>>>> openstatus(4))
>>>>>>>>
>>>>>>>>           if (openstatus(4) > 0) then
>>>>>>>>
>>>>>>>>               print *, "ibm3d_input file not present or wrong filename."
>>>>>>>>
>>>>>>>>               stop
>>>>>>>>
>>>>>>>>           end if
>>>>>>>>
>>>>>>>>           read (44,nml = solver_input)
>>>>>>>>
>>>>>>>>           read (44,nml = grid_input)
>>>>>>>>
>>>>>>>>           read (44,nml = body_input)...
>>>>>>>>
>>>>>>>>
>>>>>>>> After using PETSc 64bit, my code aborts and I realise that it is because
>>>>>>>> the values have became NaN. Strangely, it does not occur in windows with
>>>>>>>> VS2008.
>>>>>>>>
>>>>>>>> I wonder if it's a bug with the Intel Fortran compiler 2018.
>>>>>>>>
>>>>>>>> Anyone has similar experience?
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Thank you very much.
>>>>>>>>
>>>>>>>> Yours sincerely,
>>>>>>>>
>>>>>>>> ================================================
>>>>>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>>>>> Youtube research showcase:
>>>>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>>>>> ================================================
>>>>>>>>

From bsmith at mcs.anl.gov  Wed Apr  3 01:05:27 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 3 Apr 2019 06:05:27 +0000
Subject: [petsc-users] Unable to read in values thru namelist in Fortran
 after using PETSc 64bit in linux
In-Reply-To: <1c3ff9e7-ebf8-a0e6-c2e9-5d2307bdf656@gmail.com>
References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com>
	<0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov>
	<22565566-590d-7c89-390a-99557ce05382@gmail.com>
	<alpine.LFD.2.21.1809180034010.2752@asterix>
	<cdde09d2-f3eb-21c9-d0a7-fda195c17be5@gmail.com>
	<C4D68689-DDF9-4C94-9C20-FB10F2C748FD@mcs.anl.gov>
	<69d01b45-050f-2729-3015-14f3d80202d5@gmail.com>
	<686A13FC-7B4E-4F7C-84E0-3A0FCFC08FA1@mcs.anl.gov>
	<1c3ff9e7-ebf8-a0e6-c2e9-5d2307bdf656@gmail.com>
Message-ID: <43DFABEE-CC82-4BDA-A17A-611CF8911190@mcs.anl.gov>


   Based on the data below I am guess you are passing PetscInt arguments for all the arguments to MPI_ALLGATHERV. This won't work if PetscInt variables are of size 64 bits. Lets look at the arguments from https://www.mpich.org/static/docs/v3.1.x/www3/MPI_Allgather.html

int MPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                  void *recvbuf, int recvcount, MPI_Datatype recvtype,
                  MPI_Comm comm)

all he arguments labeled void* can be passed PetscInt declared arguments but the ones labeled int must be passed regular 32 bit integers (to help our own development we label all these variables as PetscMPIInt in PETSc)

   Good luck, if this does not resolve your problem please send us a small standalone problem we can run that reproduces your problem.

   Barry




> On Apr 3, 2019, at 12:48 AM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> Hi,
> 
> I just encounter a mpi_allgatherv problem when using 64bit PETSc:
> 
> call MPI_ALLGATHERV(tmp_mpi_data,counter,MPIU_INTEGER,tmp_mpi_data2,counter_global,idisp,MPIU_INTEGER,MPI_COMM_WORLD,ierr)
> 
> The error is:
> 
> Fatal error in PMPI_Allgatherv: Message truncated, error stack:
> PMPI_Allgatherv(1452).....: MPI_Allgatherv(sbuf=0x75cfdc0, scount=1119, dtype=0x4c000831, rbuf=0x75d8600, rcounts=0x7ffffffdb880, displs=0x7ffffffdb860, dtype=0x4c000831, MPI_COMM_WORLD) failed
> MPIR_Allgatherv_impl(1013): fail failed
> MPIR_Allgatherv(967)......: fail failed
> 
> MPIR_Allgatherv_intra(222): fail failed
> MPIR_Localcopy(107).......: Message truncated; 8952 bytes received but buffer size is 8864
> 
> The variables are all defined as PetscInt. The strange thing is that I did 2-3 MPI_ALLGATHERV which are exactly the same, but only the last one got problem. Should I change all these variables to PetscMPIInt?
> 
> Also, can I change all PetscInt in the code to PetscMPIInt, except if it's labeled void *?
> 
> 
> Thank you very much.
> 
> Yours sincerely,
> 
> ================================================
> TAY Wee-Beng (Zheng Weiming) ???
> Personal research webpage: http://tayweebeng.wixsite.com/website
> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
> linkedin: www.linkedin.com/in/tay-weebeng
> ================================================
> 
> On 21/9/2018 2:57 AM, Smith, Barry F. wrote:
>>    Yes, you need to go through your code and check each MPI call and make sure you use PetscMPIInt for integer arguments and PetscInt for the void* arguments and also make sure that the data type you use in the MPI calls (when communicating PetscInt) is MPIU_INT.
>> 
>>     You should not need a fancy debugger to find out the crash point. Just a basic debugger like gdb, lldb, or dbx will
>> 
>>    Barry
>> 
>> 
>>> On Sep 20, 2018, at 8:57 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> Sorry I'm still a bit confused. My 64bit code still doesn't work once I use more than 1 procs. It just aborts at some point. I'm been trying to use ARM Forge mpi debugging tool to find the error but it's a bit difficult to back trace.
>>> 
>>> So I should carefully inspect each mpi subroutine or function, is that correct?
>>> 
>>> If it's INT, then I should use PetscMPIInt. If it's labeled void *, I should use PetscInt. Is that so?
>>> 
>>> Thank you very much
>>> 
>>> Yours sincerely,
>>> 
>>> ================================================
>>> TAY Wee-Beng ??? (Zheng Weiming)
>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>> linkedin: www.linkedin.com/in/tay-weebeng
>>> ================================================
>>> 
>>> On 19/9/2018 12:49 AM, Smith, Barry F. wrote:
>>>>    PetscMPIInt  (or integer) are for all lengths passed to MPI functions; simple look at the prototypes for the MPI function you care about and it will tell you which arguments are integer.  The DATA you are passing into the MPI arrays (which are labeled void * in the manual pages) should be PetscInt.
>>>> 
>>>> 
>>>> 
>>>>> On Sep 18, 2018, at 1:39 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER?
>>>>> 
>>>>> In other words, must I assign local_array_length etc as PetscMPIInt?
>>>>> 
>>>>> call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr)
>>>>> 
>>>>> Or is it ok to change all integers from PetscInt to PetscMPIInt?
>>>>> 
>>>>> With the exception of ierr - PetscErrorCode
>>>>> 
>>>>> 
>>>>> Thank you very much.
>>>>> 
>>>>> Yours sincerely,
>>>>> 
>>>>> ================================================
>>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>> ================================================
>>>>> 
>>>>> On 18/9/2018 1:39 PM, Balay, Satish wrote:
>>>>>> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html
>>>>>> 
>>>>>> int MPI_Comm_size( MPI_Comm comm, int *size )
>>>>>> 
>>>>>> i.e there is no PetscInt here. [MPI does not know about PETSc datatypes]
>>>>>> 
>>>>>> For convinence we provide PetscMPIInt to keep track of such variables
>>>>>> [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F
>>>>>> 
>>>>>> Satish
>>>>>> 
>>>>>> On Tue, 18 Sep 2018, TAY wee-beng wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I managed to find the error appearing after using PETSc 64bit in linux -
>>>>>>> 
>>>>>>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr)
>>>>>>> 
>>>>>>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs)
>>>>>>> 
>>>>>>> Assigning num_procs as integer as the problem.
>>>>>>> 
>>>>>>> Is this supposed to be the case? Or is it a bug?
>>>>>>> 
>>>>>>> Thank you very much.
>>>>>>> 
>>>>>>> Yours sincerely,
>>>>>>> 
>>>>>>> ================================================
>>>>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>>>> Youtube research showcase:
>>>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>>>> ================================================
>>>>>>> 
>>>>>>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote:
>>>>>>>>        You can try valgrind
>>>>>>>>        http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>> 
>>>>>>>>    Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I found that I am unable to read in values thru namelist in Fortran after
>>>>>>>>> using PETSc 64bit in linux.
>>>>>>>>> 
>>>>>>>>> I have a parameter txt file which is read in using namelist in Fortran?
>>>>>>>>> 
>>>>>>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config
>>>>>>>>> ...
>>>>>>>>> 
>>>>>>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat =
>>>>>>>>> openstatus(4))
>>>>>>>>> 
>>>>>>>>>          if (openstatus(4) > 0) then
>>>>>>>>> 
>>>>>>>>>              print *, "ibm3d_input file not present or wrong filename."
>>>>>>>>> 
>>>>>>>>>              stop
>>>>>>>>> 
>>>>>>>>>          end if
>>>>>>>>> 
>>>>>>>>>          read (44,nml = solver_input)
>>>>>>>>> 
>>>>>>>>>          read (44,nml = grid_input)
>>>>>>>>> 
>>>>>>>>>          read (44,nml = body_input)...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> After using PETSc 64bit, my code aborts and I realise that it is because
>>>>>>>>> the values have became NaN. Strangely, it does not occur in windows with
>>>>>>>>> VS2008.
>>>>>>>>> 
>>>>>>>>> I wonder if it's a bug with the Intel Fortran compiler 2018.
>>>>>>>>> 
>>>>>>>>> Anyone has similar experience?
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Thank you very much.
>>>>>>>>> 
>>>>>>>>> Yours sincerely,
>>>>>>>>> 
>>>>>>>>> ================================================
>>>>>>>>> TAY Wee-Beng (Zheng Weiming) ???
>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>>>>>> Youtube research showcase:
>>>>>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>>>>>> ================================================
>>>>>>>>> 


From ncreati at inogs.it  Wed Apr  3 03:07:09 2019
From: ncreati at inogs.it (Nicola Creati)
Date: Wed, 3 Apr 2019 10:07:09 +0200
Subject: [petsc-users] petsc4py - Convert c example code to Python
In-Reply-To: <83CD5287-183D-48B0-BCF0-1CA4AF6BB85A@anl.gov>
References: <4ad0f9a1-ce21-fafb-9edf-bb1ad470f7b6@inogs.it>
	<83CD5287-183D-48B0-BCF0-1CA4AF6BB85A@anl.gov>
Message-ID: <b92de50f-1322-b113-8e0a-cda3551b1ef6@inogs.it>

Hello,
thanks to all, I used the wrong RHS and Jacobian function in my 
conversion. I modified the code and I got the right solution. I'm 
attaching the final code conversion. Someone could be interested in a 
Python version of ex13 of the TS component tutorial.

"""
Python version of the ex13.c PETSc TS component at:
https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex13.c.html
"""

import sys, petsc4py
petsc4py.init(sys.argv)

from petsc4py import PETSc
from mpi4py import MPI
import numpy as np
import math

def RHS_func(ts, t, U, F, *args):
 ??? da = ts.getDM()
 ??? localU = da.getLocalVec()

 ??? da.globalToLocal(U, localU)

 ??? mx, my = da.getSizes()

 ??? hx, hy = [1.0/(m-1) for m in [mx, my]]
 ??? sx = 1.0/(hx*hx)
 ??? sy = 1.0/(hy*hy)

 ??? uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
 ??? f = F.getArray(readonly=0).reshape(8, 8, order='C')

 ??? (xs, xm), (ys, ym) = da.getRanges()
 ??? for j in range(ys, ym):
 ??????? for i in range(xs, xm):
 ??????????? if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
 ??????????????? f[i,j] = uarray[i,j]
 ??????????????? continue
 ??????????? u = uarray[i,j]
 ??????????? uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
 ??????????? uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
 ??????????? f[i, j] = uxx + uyy
 ??? F.assemble()

def Jacobian_func(ts, t, U, A, B, *args):
 ??? mx, my = da.getSizes()
 ??? hx = 1.0/(mx-1)
 ??? hy = 1.0/(my-1)
 ??? sx = 1.0/(hx*hx)
 ??? sy = 1.0/(hy*hy)

 ??? B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
 ??? B.zeroEntries()

 ??? (i0, i1), (j0, j1) = da.getRanges()
 ??? row = PETSc.Mat.Stencil()
 ??? col = PETSc.Mat.Stencil()

 ??? for i in range(j0, j1):
 ??????? for j in range(i0, i1):
 ??????????? row.index = (i,j)
 ??????????? row.field = 0
 ??????????? if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
 ??????????????? B.setValueStencil(row, row, 1.0)
 ??????????? else:
 ??????????????? for index, value in [
 ??????????????????? ((i-1, j),?? sx),
 ??????????????????? ((i+1, j),?? sx),
 ??????????????????? ((i,?? j-1), sy),
 ??????????????????? ((i, j+1), sy),
 ??????????????????? ((i,?? j),? -2*sx - 2*sy)]:
 ??????????????????? col.index = index
 ??????????????????? col.field = 0
 ??????????????????? B.setValueStencil(row, col, value)

 ??? B.assemble()
 ??? if A != B: B.assemble()

 ??? return PETSc.Mat.Structure.SAME_NONZERO_PATTERN

def make_initial_solution(da, U, c):
 ??? mx, my = da.getSizes()
 ??? hx = 1.0/(mx-1)
 ??? hy = 1.0/(my-1)
 ??? (xs, xm), (ys, ym) = da.getRanges()

 ??? u = U.getArray(readonly=0).reshape(8, 8, order='C')

 ??? for j in range(ys, ym):
 ??????? y = j*hy
 ??????? for i in range(xs, xm):
 ??????????? x = i*hx
 ??????????? r = math.sqrt((x-0.5)**2+(y-0.5)**2)
 ??????????? if r < (0.125):
 ??????????????? u[i, j] = math.exp(c*r*r*r)
 ??????????? else:
 ??????????????? u[i, j] = 0.0
 ??? U.assemble()

nx = 8
ny = 8
da = PETSc.DMDA().create([nx, ny], stencil_type= 
PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)

da.setFromOptions()
da.setUp()

u = da.createGlobalVec()
f = u.duplicate()

c = -30.0

ts = PETSc.TS().create()
ts.setDM(da)
ts.setType(ts.Type.BEULER)

ts.setRHSFunction(RHS_func, f)

da.setMatType('aij')
J = da.createMat()
ts.setRHSJacobian(Jacobian_func, J, J)

ftime = 1.0
ts.setMaxTime(ftime)
ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)

make_initial_solution(da, u, c)
dt = 0.01

ts.setTimeStep(dt)
ts.setFromOptions()
ts.solve(u)

ftime = ts.getSolveTime()
steps = ts.getStepNumber()

Cheers.

Nicola


On 02/04/2019 16:54, Zhang, Hong wrote:
> Your python implementation of the residual function and the Jacobian function is intended for solving ODEs in the form x' = f(t,x). So you should use ts.setRHSFunction() and ts.setRHSJacobian() instead of the implicit version.
>
> Hong
>
>> On Apr 2, 2019, at 3:18 AM, Nicola Creati via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hello, I have run the two codes using: -pc_type lu -ts_monitor -snes_monitor -ksp_monitor -ts_max_steps 5. The codes start to diverge after the first time step:
>> Python run :
>>
>> 0 TS dt 0.01 time 0.
>>      0 SNES Function norm 2.327405179696e+02
>>        0 KSP Residual norm 1.939100379240e+00
>>        1 KSP Residual norm 1.013760235597e-15
>>      1 SNES Function norm 9.538686413612e-14
>> 1 TS dt 0.01 time 0.01
>>      0 SNES Function norm 8.565210784140e-14
>>        0 KSP Residual norm 1.266678408555e-15
>>        1 KSP Residual norm 2.663404219322e-31
>>      1 SNES Function norm 5.528402779439e-29
>> 2 TS dt 0.01 time 0.02
>>      0 SNES Function norm 5.829656072531e-29
>>        0 KSP Residual norm 3.805606380832e-31
>>        1 KSP Residual norm 3.283104975114e-46
>>      1 SNES Function norm 1.867771204856e-44
>> 3 TS dt 0.01 time 0.03
>>      0 SNES Function norm 1.644541571708e-44
>>        0 KSP Residual norm 1.969573456719e-46
>>        1 KSP Residual norm 1.297976290541e-61
>>      1 SNES Function norm 1.780215821230e-59
>> 4 TS dt 0.01 time 0.04
>>      0 SNES Function norm 1.700894451833e-59
>> 5 TS dt 0.01 time 0.05
>>
>> ex13.c run:
>>
>> 0 TS dt 0.01 time 0.
>>      0 SNES Function norm 2.327405179696e+02
>>        0 KSP Residual norm 9.158498731719e-01
>>        1 KSP Residual norm 5.129005976821e-16
>>      1 SNES Function norm 1.141297389434e-13
>> 1 TS dt 0.01 time 0.01
>>      0 SNES Function norm 9.158498731719e+01
>>        0 KSP Residual norm 4.156347391374e-01
>>        1 KSP Residual norm 1.996081836894e-16
>>      1 SNES Function norm 3.609761734594e-14
>> 2 TS dt 0.01 time 0.02
>>      0 SNES Function norm 4.156347391374e+01
>>        0 KSP Residual norm 2.203298795311e-01
>>        1 KSP Residual norm 9.047356740098e-17
>>      1 SNES Function norm 3.292648977280e-14
>> 3 TS dt 0.01 time 0.03
>>      0 SNES Function norm 2.203298795311e+01
>>        0 KSP Residual norm 1.367458644503e-01
>>        1 KSP Residual norm 5.055030661919e-17
>>      1 SNES Function norm 1.272304808486e-14
>> 4 TS dt 0.01 time 0.04
>>      0 SNES Function norm 1.367458644503e+01
>>        0 KSP Residual norm 9.695393464180e-02
>>        1 KSP Residual norm 1.918401734795e-17
>>      1 SNES Function norm 1.130247628824e-14
>> 5 TS dt 0.01 time 0.05
>>
>> I rechecked the Python code against the C one but I did not find the source of the issue.
>>
>> # Example 13 petsc TS
>> import sys, petsc4py
>> petsc4py.init(sys.argv)
>>
>> from petsc4py import PETSc
>> from mpi4py import MPI
>> import numpy as np
>> import math
>>
>> def RHS_func(ts, t, X, xdot, F):
>>      da = ts.getDM()
>>      localU = da.getLocalVec()
>>
>>      da.globalToLocal(X, localU)
>>
>>      mx, my = da.getSizes()
>>
>>      hx, hy = [1.0/(m-1) for m in [mx, my]]
>>      sx = 1.0/(hx*hx)
>>      sy = 1.0/(hy*hy)
>>
>>      uarray = localU.getArray(readonly=1).reshape(8, 8, order='C')
>>      f = F.getArray(readonly=0).reshape(8, 8, order='C')
>>
>>      (xs, xm), (ys, ym) = da.getRanges()
>>      for j in range(ys, ym):
>>          for i in range(xs, xm):
>>              if i == 0 or j == 0 or i == (mx-1) or j == (my-1):
>>                  f[i,j] = uarray[i,j]
>>                  continue
>>              u = uarray[i,j]
>>              uxx = (-2.0 * u + uarray[i, j-1] + uarray[i, j+1]) * sx
>>              uyy = (-2.0 * u + uarray[i-1, j] + uarray[i+1, j])* sy
>>              f[i, j] = uxx + uyy
>>      F.assemble()
>>      #PETSc.Log.logFlops(11.0*xm*ym)
>>
>>
>> def Jacobian_func(ts, t, x, xdot, a, A, B):
>>      mx, my = da.getSizes()
>>      hx = 1.0/(mx-1)
>>      hy = 1.0/(my-1)
>>      sx = 1.0/(hx*hx)
>>      sy = 1.0/(hy*hy)
>>
>>      B.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False)
>>      B.zeroEntries()
>>
>>      (i0, i1), (j0, j1) = da.getRanges()
>>      row = PETSc.Mat.Stencil()
>>      col = PETSc.Mat.Stencil()
>>
>>      for i in range(j0, j1):
>>          for j in range(i0, i1):
>>              row.index = (i,j)
>>              row.field = 0
>>              if i == 0 or j== 0 or i==(my-1) or j==(mx-1):
>>                  B.setValueStencil(row, row, 1.0)
>>              else:
>>                  for index, value in [
>>                      ((i-1, j),   sx),
>>                      ((i+1, j),   sx),
>>                      ((i,   j-1), sy),
>>                      ((i, j+1), sy),
>>                      ((i,   j),  -2*sx - 2*sy)]:
>>                      col.index = index
>>                      col.field = 0
>>                      B.setValueStencil(row, col, value)
>>
>>      B.assemble()
>>      #B.view()
>>      if A != B: B.assemble()
>>
>>      return PETSc.Mat.Structure.SAME_NONZERO_PATTERN
>>
>> def make_initial_solution(da, U, c):
>>      mx, my = da.getSizes()
>>      hx = 1.0/(mx-1)
>>      hy = 1.0/(my-1)
>>      (xs, xm), (ys, ym) = da.getRanges()
>>
>>      u = U.getArray(readonly=0).reshape(8, 8, order='C')
>>
>>      for j in range(ys, ym):
>>          y = j*hy
>>          for i in range(xs, xm):
>>              x = i*hx
>>              r = math.sqrt((x-0.5)**2+(y-0.5)**2)
>>              if r < (0.125):
>>                  u[i, j] = math.exp(c*r*r*r)
>>              else:
>>                  u[i, j] = 0.0
>>      U.assemble()
>>      #U.view()
>>
>>
>> def monitor(ts, i, t, x):
>>      xx = x[:].tolist()
>>      history.append((i, t, xx))
>>
>>
>> history = []
>> nx = 8
>> ny = 8
>> da = PETSc.DMDA().create([nx, ny], stencil_type= PETSc.DA.StencilType.STAR, stencil_width=1, dof=1)
>>
>> da.setFromOptions()
>> da.setUp()
>>
>> u = da.createGlobalVec()
>> f = u.duplicate()
>>
>> c = -30.0
>>
>> ts = PETSc.TS().create()
>> ts.setDM(da)
>> ts.setType(ts.Type.BEULER)
>>
>> ts.setIFunction(RHS_func, f)
>>
>> da.setMatType('aij')
>> J = da.createMat()
>> ts.setIJacobian(Jacobian_func, J, J)
>>
>> ftime = 1.0
>> ts.setMaxTime(ftime)
>> ts.setExactFinalTime(PETSc.TS.ExactFinalTime.STEPOVER)
>>
>> make_initial_solution(da, u, c)
>> dt = 0.01
>>
>> ts.setMonitor(monitor)
>> ts.setTimeStep(dt)
>> ts.setFromOptions()
>> ts.solve(u)
>>
>> ftime = ts.getSolveTime()
>> steps = ts.getStepNumber()
>>
>> Thanks.
>>
>> Nicola
>>
>>
>> -- 
>>
>> Nicola Creati
>> Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
>> Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
>> CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
>> Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
>> off.   +39 040 2140 213
>> fax.   +39 040 327307
>>
>> _____________________________________________________________________
>> This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.
>>

-- 

Nicola Creati
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it
Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department
CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/
Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati at ogs.trieste.it
off.   +39 040 2140 213
fax.   +39 040 327307

_____________________________________________________________________
This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario.


From dalcinl at gmail.com  Wed Apr  3 03:39:56 2019
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 3 Apr 2019 11:39:56 +0300
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
Message-ID: <CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>

IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the
signedness of plain `char` is implementation (or platform?) dependent.
 I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you
should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that
MPI_SIGNED_CHAR is from MPI 2.0.

On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi All,
>
> There were some error messages when using PetscSFReduceBegin with
> MPI_CHAR.
>
> ierr =
> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>
>
> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR?
> If not, should we document somewhere?
>
> Thanks
>
> Fande,
>
>
> [*0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> *[0]PETSC ERROR: No support for this operation for this object type*
> *[0]PETSC ERROR: No support for type size not divisible by 4*
> *[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
> *[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>  GIT Date: 2019-04-02 17:37:18 -0600*
> *[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> *[1]PETSC ERROR: No support for this operation for this object type*
> *[1]PETSC ERROR: No support for type size not divisible by 4*
> *[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
> *[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e
>  GIT Date: 2019-04-02 17:37:18 -0600*
> *[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once
> named fn605731.local by kongf Tue Apr  2 21:48:41 2019*
> *[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
> --download-parmetis=1 --download-superlu_dist=1
> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
> --download-party --download-chaco --with-cxx-dialect=C++11*
> *[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named
> fn605731.local by kongf Tue Apr  2 21:48:41 2019*
> *[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes
> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
> --download-parmetis=1 --download-superlu_dist=1
> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
> --download-party --download-chaco --with-cxx-dialect=C++11*
> *[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
> *[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
> *[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[0]PETSC ERROR: #7 MatPtAP() line 9429 in
> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
> *[0]PETSC ERROR: #8 main() line 58 in
> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
> *[0]PETSC ERROR: PETSc Option Table entries:*
> *[0]PETSC ERROR: -matptap_via allatonce*
> *[0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------*
> *[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
> *[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
> *[1]PETSC ERROR: #7 MatPtAP() line 9429 in
> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
> *[1]PETSC ERROR: #8 main() line 58 in
> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
> *[1]PETSC ERROR: PETSc Option Table entries:*
> *[1]PETSC ERROR: -matptap_via allatonce*
> *[1]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------*
>
> *--------------------------------------------------------------------------*
> *MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD*
> *with errorcode 56.*
>
> *NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.*
> *You may or may not see output from other processes, depending on*
> *exactly when Open MPI kills them.*
>
> *--------------------------------------------------------------------------*
> *[fn605731.local:78133] 1 more process has sent help message
> help-mpi-api.txt / mpi-abort*
> *[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0
> to see all help / error messages*
>


-- 
Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190403/51320ebe/attachment.html>

From griesser.jan at googlemail.com  Wed Apr  3 06:05:21 2019
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 3 Apr 2019 13:05:21 +0200
Subject: [petsc-users] Integrate command line arguments -bv_type vecs
 -bv_type mat to python code
Message-ID: <CAB_-EyVGT2AMcFs9M9+Q36fCxVbMzbKk+Z4mF12WgRrUsoBsZg@mail.gmail.com>

Hello, everybody,
I use Splec4py to solve the lowest eigenvalues and corresponding
eigenvectors of systems up to 20 Mio x 20 Mio. My problem so far was that
the base was too big. On advice here in the forum I always added -bv_type
vecs -bv_type mat to the command line, what solved my problems. I don't
always want to execute these two commands on the command line, but I want
to define them directly in the python code before E.solve() is called.
However, I'm not quite sure how to call these functions correctly in
Python.
Currently I do it like this
 bv = SLEPc.BV().create()
bv.setType(VECS),
bv.setType(Mat), E.setBV(bv)

Could you tell me if that's correct?
Thank you very much in advance!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190403/9dd4a61a/attachment-0001.html>

From jroman at dsic.upv.es  Wed Apr  3 06:12:13 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 3 Apr 2019 13:12:13 +0200
Subject: [petsc-users] Integrate command line arguments -bv_type vecs
 -bv_type mat to python code
In-Reply-To: <CAB_-EyVGT2AMcFs9M9+Q36fCxVbMzbKk+Z4mF12WgRrUsoBsZg@mail.gmail.com>
References: <CAB_-EyVGT2AMcFs9M9+Q36fCxVbMzbKk+Z4mF12WgRrUsoBsZg@mail.gmail.com>
Message-ID: <C5142B5D-FB3E-4E39-BB70-FA5DE0BACBE3@dsic.upv.es>

You could just do

  E.getBV().setType(SLEPc.BV.Type.VECS)

before E.setFromOptions()

Jose


> El 3 abr 2019, a las 13:05, Jan Grie?er via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hello, everybody,
> I use Splec4py to solve the lowest eigenvalues and corresponding eigenvectors of systems up to 20 Mio x 20 Mio. My problem so far was that the base was too big. On advice here in the forum I always added -bv_type vecs -bv_type mat to the command line, what solved my problems. I don't always want to execute these two commands on the command line, but I want to define them directly in the python code before E.solve() is called. However, I'm not quite sure how to call these functions correctly in Python. 
> Currently I do it like this
>  bv = SLEPc.BV().create()
> bv.setType(VECS),
> bv.setType(Mat), E.setBV(bv)
> 
> Could you tell me if that's correct? 
> Thank you very much in advance!


From jed at jedbrown.org  Wed Apr  3 17:09:40 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 03 Apr 2019 16:09:40 -0600
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<3daa411c-d2c4-53d3-ff7e-c14429b40e49@idris.fr>
	<CADOhEh7u95=X8wHuaVhUXTK=Kj62Jm+COJOdPfTyXB_ycvQcZA@mail.gmail.com>
	<7b104336-a067-e679-23ec-2a89e0ba9bc4@idris.fr>
	<CADOhEh5MHFQOoHF24hOWhoAiLnqYuZO29hwKzHmMndicF1xj0g@mail.gmail.com>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
Message-ID: <877ecaeo63.fsf@jedbrown.org>

Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi all,
>
> for your information, you'll find attached the comparison of the weak
> memory scalings when using :
>
> - PETSc 3.6.4 (reference)
> - PETSc 3.10.4 without specific options
> - PETSc 3.10.4 with the three scalability options you mentionned
>
> Using the scalability options does improve the memory scaling. However,
> the 3.6 version still has a better one...

Yes, this still looks significant.  Is this an effect we can still
reproduce with a PETSc example and/or using a memory profiler (such as
massif or gperftools)?  I think it's important for us to narrow down
what causes this difference (looks like almost 2x on your 1e8 problem
size) so we can fix.

From bsmith at mcs.anl.gov  Wed Apr  3 20:39:24 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 4 Apr 2019 01:39:24 +0000
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
Message-ID: <8FC532D6-E044-42AE-95C7-F2FC29089D0C@anl.gov>


  It is probably not written down anywhere (where would we write it?) nor does ./configure check it but the consensus is that PETSc now requires MPI 2.0 or later.  So developers should feel free to use any MPI 2.0 feature without guards and if you stumble across older code that still has guards/configure tests for 2.0 features those may be removed.

   Barry

  On the other hand we still require C89* code so no one would make the mistake of thinking PETSc is cutting edge ;)

  * One word -- Microsoft


> On Apr 3, 2019, at 3:39 AM, Lisandro Dalcin via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the signedness of plain `char` is implementation (or platform?) dependent.
>  I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that MPI_SIGNED_CHAR is from MPI 2.0.
> 
> On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi All,
> 
> There were some error messages when using PetscSFReduceBegin with MPI_CHAR. 
> 
> ierr = PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
> 
> 
> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR? If not, should we document somewhere?
> 
> Thanks
> 
> Fande,
> 
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: No support for type size not divisible by 4
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: No support for this operation for this object type
> [1]PETSC ERROR: No support for type size not divisible by 4
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
> [1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
> [1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
> [1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> [1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> [1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
> [0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
> [0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> [0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> [0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
> [0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
> [0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [0]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -matptap_via allatonce
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
> [1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
> [1]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
> [1]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: -matptap_via allatonce
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 56.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [fn605731.local:78133] 1 more process has sent help message help-mpi-api.txt / mpi-abort
> [fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> 
> 
> -- 
> Lisandro Dalcin
> ============
> Research Scientist
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/


From jczhang at mcs.anl.gov  Wed Apr  3 22:15:07 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 4 Apr 2019 03:15:07 +0000
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
Message-ID: <CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>


On Wed, Apr 3, 2019 at 3:41 AM Lisandro Dalcin via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the signedness of plain `char` is implementation (or platform?) dependent.
 I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that MPI_SIGNED_CHAR is from MPI 2.0.

MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations"
So Fande's code and Jed's branch have problems. To fix that, we have to add support for signed char, unsigned char, and char in PetscSF.  The first two types support add, mult, logical and bitwise operations. The last is a dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would raise error on Fande's code. I can come up with a fix tomorrow.


On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi All,

There were some error messages when using PetscSFReduceBegin with MPI_CHAR.

ierr = PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);


My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR? If not, should we document somewhere?

Thanks

Fande,


[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: No support for type size not divisible by 4
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: No support for this operation for this object type
[1]PETSC ERROR: No support for type size not divisible by 4
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[0]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -matptap_via allatonce
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[1]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
[1]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -matptap_via allatonce
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 56.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[fn605731.local:78133] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages


--
Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/07d80737/attachment-0001.html>

From fdkong.jd at gmail.com  Wed Apr  3 22:28:52 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Wed, 3 Apr 2019 21:28:52 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
Message-ID: <C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>

Thanks for the reply.  It is not necessary for me to use MPI_SUM.  I think the better choice is MPIU_REPLACE. Doesn?t MPIU_REPLACE work for any mpi_datatype?

Fande 


> On Apr 3, 2019, at 9:15 PM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
> 
> 
>> On Wed, Apr 3, 2019 at 3:41 AM Lisandro Dalcin via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the signedness of plain `char` is implementation (or platform?) dependent.
>>  I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that MPI_SIGNED_CHAR is from MPI 2.0.
> 
> MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations"
> So Fande's code and Jed's branch have problems. To fix that, we have to add support for signed char, unsigned char, and char in PetscSF.  The first two types support add, mult, logical and bitwise operations. The last is a dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would raise error on Fande's code. I can come up with a fix tomorrow.
>  
>> 
>>> On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> Hi All,
>>> 
>>> There were some error messages when using PetscSFReduceBegin with MPI_CHAR. 
>>> 
>>> ierr = PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>>> 
>>> 
>>> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR? If not, should we document somewhere?
>>> 
>>> Thanks
>>> 
>>> Fande,
>>> 
>>> 
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: No support for type size not divisible by 4
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [1]PETSC ERROR: No support for this operation for this object type
>>> [1]PETSC ERROR: No support for type size not divisible by 4
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
>>> [1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
>>> [1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
>>> [1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> [1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> [1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
>>> [0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
>>> [0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> [0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> [0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>> [0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
>>> [0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>> [0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>> [0]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -matptap_via allatonce
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>>> [1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
>>> [1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>> [1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>> [1]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
>>> [1]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
>>> [1]PETSC ERROR: PETSc Option Table entries:
>>> [1]PETSC ERROR: -matptap_via allatonce
>>> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>> with errorcode 56.
>>> 
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>> --------------------------------------------------------------------------
>>> [fn605731.local:78133] 1 more process has sent help message help-mpi-api.txt / mpi-abort
>>> [fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
>> 
>> 
>> -- 
>> Lisandro Dalcin
>> ============
>> Research Scientist
>> Extreme Computing Research Center (ECRC)
>> King Abdullah University of Science and Technology (KAUST)
>> http://ecrc.kaust.edu.sa/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190403/d30013f4/attachment.html>

From jczhang at mcs.anl.gov  Wed Apr  3 22:40:35 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 4 Apr 2019 03:40:35 +0000
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
	<C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
Message-ID: <CA+MQGp8cG68WRJwXQSi03oRe3_rFDZ+nm7MGEc93HAa1RcWugw@mail.gmail.com>



On Wed, Apr 3, 2019, 10:29 PM Fande Kong <fdkong.jd at gmail.com<mailto:fdkong.jd at gmail.com>> wrote:
Thanks for the reply.  It is not necessary for me to use MPI_SUM.  I think the better choice is MPIU_REPLACE. Doesn?t MPIU_REPLACE work for any mpi_datatype?
Yes.
Fande


On Apr 3, 2019, at 9:15 PM, Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:


On Wed, Apr 3, 2019 at 3:41 AM Lisandro Dalcin via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the signedness of plain `char` is implementation (or platform?) dependent.
 I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that MPI_SIGNED_CHAR is from MPI 2.0.

MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations"
So Fande's code and Jed's branch have problems. To fix that, we have to add support for signed char, unsigned char, and char in PetscSF.  The first two types support add, mult, logical and bitwise operations. The last is a dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would raise error on Fande's code. I can come up with a fix tomorrow.


On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi All,

There were some error messages when using PetscSFReduceBegin with MPI_CHAR.

ierr = PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);


My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR? If not, should we document somewhere?

Thanks

Fande,


[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: No support for type size not divisible by 4
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: No support for this operation for this object type
[1]PETSC ERROR: No support for type size not divisible by 4
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
[1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
[0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[0]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -matptap_via allatonce
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------
[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
[1]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
[1]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -matptap_via allatonce
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 56.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[fn605731.local:78133] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages


--
Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/7795441e/attachment-0001.html>

From davelee2804 at gmail.com  Thu Apr  4 03:27:00 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Thu, 4 Apr 2019 19:27:00 +1100
Subject: [petsc-users] testing for and removing a null space using JFNK
Message-ID: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>

Hello PETSc,

I'm attempting to solve a JFNK problem for a system where I only have a
function to compute the residual, but no matrix.

I wanted to know if there exists functionality in PETSc to do the following:

1) approximate a null space from a set of Krylov vectors

2) remove such a null space if it exists

I'm vaguely familiar with the MatNullSpaceCreate/Remove() functionality,
however I don't know the precise form of a null space, so I don't have a
set of vectors I can assemble and pass to this.

Cheers, Dave.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/f3ed3fbb/attachment.html>

From mfadams at lbl.gov  Thu Apr  4 04:26:03 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 4 Apr 2019 05:26:03 -0400
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
Message-ID: <CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>

The Krylov space can not see the null space (by definition) and so getting
a useful near null space from it is not likely.

Getting a null space is a hard problem and bootstrap AMG methods, for
instance, are developed to try to do that. This is an advanced research
topic.

You really want to know your null space, what kind of equations do you have?

Mark

On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello PETSc,
>
> I'm attempting to solve a JFNK problem for a system where I only have a
> function to compute the residual, but no matrix.
>
> I wanted to know if there exists functionality in PETSc to do the
> following:
>
> 1) approximate a null space from a set of Krylov vectors
>
> 2) remove such a null space if it exists
>
> I'm vaguely familiar with the MatNullSpaceCreate/Remove() functionality,
> however I don't know the precise form of a null space, so I don't have a
> set of vectors I can assemble and pass to this.
>
> Cheers, Dave.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/d6da6c84/attachment.html>

From griesser.jan at googlemail.com  Thu Apr  4 05:23:12 2019
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Thu, 4 Apr 2019 12:23:12 +0200
Subject: [petsc-users] Integrate command line arguments -bv_type vecs
 -bv_type mat to python code
In-Reply-To: <C5142B5D-FB3E-4E39-BB70-FA5DE0BACBE3@dsic.upv.es>
References: <CAB_-EyVGT2AMcFs9M9+Q36fCxVbMzbKk+Z4mF12WgRrUsoBsZg@mail.gmail.com>
	<C5142B5D-FB3E-4E39-BB70-FA5DE0BACBE3@dsic.upv.es>
Message-ID: <CAB_-EyXyM+LwzyZ6zieJaP9=HGZq3k-+yrNObveDfE7xZOdaYA@mail.gmail.com>

It works. Thank you very much!!

Am Mi., 3. Apr. 2019 um 13:12 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es
>:

> You could just do
>
>   E.getBV().setType(SLEPc.BV.Type.VECS)
>
> before E.setFromOptions()
>
> Jose
>
>
> > El 3 abr 2019, a las 13:05, Jan Grie?er via petsc-users <
> petsc-users at mcs.anl.gov> escribi?:
> >
> > Hello, everybody,
> > I use Splec4py to solve the lowest eigenvalues and corresponding
> eigenvectors of systems up to 20 Mio x 20 Mio. My problem so far was that
> the base was too big. On advice here in the forum I always added -bv_type
> vecs -bv_type mat to the command line, what solved my problems. I don't
> always want to execute these two commands on the command line, but I want
> to define them directly in the python code before E.solve() is called.
> However, I'm not quite sure how to call these functions correctly in
> Python.
> > Currently I do it like this
> >  bv = SLEPc.BV().create()
> > bv.setType(VECS),
> > bv.setType(Mat), E.setBV(bv)
> >
> > Could you tell me if that's correct?
> > Thank you very much in advance!
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/a2e8b2ca/attachment.html>

From mfadams at lbl.gov  Thu Apr  4 06:31:49 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 4 Apr 2019 07:31:49 -0400
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
Message-ID: <CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>

[keep on list]

On Thu, Apr 4, 2019 at 7:08 AM Dave Lee <davelee2804 at gmail.com> wrote:

> Hi Mark,
>
> Thanks for responding. My brief scan of the literature suggested that
> there are some methods out there to approximate the null space using SVD
> methods, but I wasn't sure how mature these methods were, or if PETSc had
> some capability in this regard.
>

SVDs will do it but they are expensive.


>
> My problem is to wrap a Newton solver around an existing incompressible
> Navier Stokes solver in order to iterate over the NS solution so as to
> determine periodic structures within a weakly nonlinear flow.
>

Other team members have more experience with INS. We have a lot of
capabilities for this.

You could start by looking at:

https://www.mcs.anl.gov/petsc/documentation/tutorials/HandsOnExercise.html#example4



> I think there is a null space within my problem due to the
> incompressibility constraint (ie: the linear dependence of one velocity
> component on the others). I have some other ideas on how this may be
> removed before assembling the residual, so hopefully I can fix this at the
> level of the residual vector assembly before the problem manifests at the
> level of the Krylov vectors.
>
> Thanks again for clarifying.
>
> Cheers, Dave.
>
> On Thu, Apr 4, 2019 at 8:26 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> The Krylov space can not see the null space (by definition) and so
>> getting a useful near null space from it is not likely.
>>
>> Getting a null space is a hard problem and bootstrap AMG methods, for
>> instance, are developed to try to do that. This is an advanced research
>> topic.
>>
>> You really want to know your null space, what kind of equations do you
>> have?
>>
>> Mark
>>
>> On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hello PETSc,
>>>
>>> I'm attempting to solve a JFNK problem for a system where I only have a
>>> function to compute the residual, but no matrix.
>>>
>>> I wanted to know if there exists functionality in PETSc to do the
>>> following:
>>>
>>> 1) approximate a null space from a set of Krylov vectors
>>>
>>> 2) remove such a null space if it exists
>>>
>>> I'm vaguely familiar with the MatNullSpaceCreate/Remove() functionality,
>>> however I don't know the precise form of a null space, so I don't have a
>>> set of vectors I can assemble and pass to this.
>>>
>>> Cheers, Dave.
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/c9b493c0/attachment.html>

From davelee2804 at gmail.com  Thu Apr  4 06:34:48 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Thu, 4 Apr 2019 22:34:48 +1100
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
Message-ID: <CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>

Thanks Mark,

I already have the Navier Stokes solver. My issue is wrapping it in a JFNK
solver to find the periodic solutions. I will keep reading up on SVD
approaches, there may be some capability for something like this in SLEPc.

Cheers, Dave.

On Thu, Apr 4, 2019 at 10:32 PM Mark Adams <mfadams at lbl.gov> wrote:

> [keep on list]
>
> On Thu, Apr 4, 2019 at 7:08 AM Dave Lee <davelee2804 at gmail.com> wrote:
>
>> Hi Mark,
>>
>> Thanks for responding. My brief scan of the literature suggested that
>> there are some methods out there to approximate the null space using SVD
>> methods, but I wasn't sure how mature these methods were, or if PETSc had
>> some capability in this regard.
>>
>
> SVDs will do it but they are expensive.
>
>
>>
>> My problem is to wrap a Newton solver around an existing incompressible
>> Navier Stokes solver in order to iterate over the NS solution so as to
>> determine periodic structures within a weakly nonlinear flow.
>>
>
> Other team members have more experience with INS. We have a lot of
> capabilities for this.
>
> You could start by looking at:
>
>
> https://www.mcs.anl.gov/petsc/documentation/tutorials/HandsOnExercise.html#example4
>
>
>
>> I think there is a null space within my problem due to the
>> incompressibility constraint (ie: the linear dependence of one velocity
>> component on the others). I have some other ideas on how this may be
>> removed before assembling the residual, so hopefully I can fix this at the
>> level of the residual vector assembly before the problem manifests at the
>> level of the Krylov vectors.
>>
>> Thanks again for clarifying.
>>
>> Cheers, Dave.
>>
>> On Thu, Apr 4, 2019 at 8:26 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> The Krylov space can not see the null space (by definition) and so
>>> getting a useful near null space from it is not likely.
>>>
>>> Getting a null space is a hard problem and bootstrap AMG methods, for
>>> instance, are developed to try to do that. This is an advanced research
>>> topic.
>>>
>>> You really want to know your null space, what kind of equations do you
>>> have?
>>>
>>> Mark
>>>
>>> On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hello PETSc,
>>>>
>>>> I'm attempting to solve a JFNK problem for a system where I only have a
>>>> function to compute the residual, but no matrix.
>>>>
>>>> I wanted to know if there exists functionality in PETSc to do the
>>>> following:
>>>>
>>>> 1) approximate a null space from a set of Krylov vectors
>>>>
>>>> 2) remove such a null space if it exists
>>>>
>>>> I'm vaguely familiar with the MatNullSpaceCreate/Remove()
>>>> functionality, however I don't know the precise form of a null space, so I
>>>> don't have a set of vectors I can assemble and pass to this.
>>>>
>>>> Cheers, Dave.
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/2bad9783/attachment-0001.html>

From mfadams at lbl.gov  Thu Apr  4 06:45:16 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 4 Apr 2019 07:45:16 -0400
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
Message-ID: <CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>

On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:

> Thanks Mark,
>
> I already have the Navier Stokes solver. My issue is wrapping it in a JFNK
> solver to find the periodic solutions. I will keep reading up on SVD
> approaches, there may be some capability for something like this in SLEPc.
>

Yes, SLEPc will give you parallel eigen solvers, etc.


>
> Cheers, Dave.
>
> On Thu, Apr 4, 2019 at 10:32 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> [keep on list]
>>
>> On Thu, Apr 4, 2019 at 7:08 AM Dave Lee <davelee2804 at gmail.com> wrote:
>>
>>> Hi Mark,
>>>
>>> Thanks for responding. My brief scan of the literature suggested that
>>> there are some methods out there to approximate the null space using SVD
>>> methods, but I wasn't sure how mature these methods were, or if PETSc had
>>> some capability in this regard.
>>>
>>
>> SVDs will do it but they are expensive.
>>
>>
>>>
>>> My problem is to wrap a Newton solver around an existing incompressible
>>> Navier Stokes solver in order to iterate over the NS solution so as to
>>> determine periodic structures within a weakly nonlinear flow.
>>>
>>
>> Other team members have more experience with INS. We have a lot of
>> capabilities for this.
>>
>> You could start by looking at:
>>
>>
>> https://www.mcs.anl.gov/petsc/documentation/tutorials/HandsOnExercise.html#example4
>>
>>
>>
>>> I think there is a null space within my problem due to the
>>> incompressibility constraint (ie: the linear dependence of one velocity
>>> component on the others). I have some other ideas on how this may be
>>> removed before assembling the residual, so hopefully I can fix this at the
>>> level of the residual vector assembly before the problem manifests at the
>>> level of the Krylov vectors.
>>>
>>> Thanks again for clarifying.
>>>
>>> Cheers, Dave.
>>>
>>> On Thu, Apr 4, 2019 at 8:26 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> The Krylov space can not see the null space (by definition) and so
>>>> getting a useful near null space from it is not likely.
>>>>
>>>> Getting a null space is a hard problem and bootstrap AMG methods, for
>>>> instance, are developed to try to do that. This is an advanced research
>>>> topic.
>>>>
>>>> You really want to know your null space, what kind of equations do you
>>>> have?
>>>>
>>>> Mark
>>>>
>>>> On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hello PETSc,
>>>>>
>>>>> I'm attempting to solve a JFNK problem for a system where I only have
>>>>> a function to compute the residual, but no matrix.
>>>>>
>>>>> I wanted to know if there exists functionality in PETSc to do the
>>>>> following:
>>>>>
>>>>> 1) approximate a null space from a set of Krylov vectors
>>>>>
>>>>> 2) remove such a null space if it exists
>>>>>
>>>>> I'm vaguely familiar with the MatNullSpaceCreate/Remove()
>>>>> functionality, however I don't know the precise form of a null space, so I
>>>>> don't have a set of vectors I can assemble and pass to this.
>>>>>
>>>>> Cheers, Dave.
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/c88d068c/attachment.html>

From davelee2804 at gmail.com  Thu Apr  4 06:46:28 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Thu, 4 Apr 2019 22:46:28 +1100
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
	<CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
Message-ID: <CAC4fGOHsiiYnh0f73rwy0y=fFqVUFVn3fRrDaYQL6wL5-Cz9Xw@mail.gmail.com>

Thanks for your help Mark.

Cheers, Dave.

On Thu, Apr 4, 2019 at 10:45 PM Mark Adams <mfadams at lbl.gov> wrote:

>
>
> On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:
>
>> Thanks Mark,
>>
>> I already have the Navier Stokes solver. My issue is wrapping it in a
>> JFNK solver to find the periodic solutions. I will keep reading up on SVD
>> approaches, there may be some capability for something like this in SLEPc.
>>
>
> Yes, SLEPc will give you parallel eigen solvers, etc.
>
>
>>
>> Cheers, Dave.
>>
>> On Thu, Apr 4, 2019 at 10:32 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> [keep on list]
>>>
>>> On Thu, Apr 4, 2019 at 7:08 AM Dave Lee <davelee2804 at gmail.com> wrote:
>>>
>>>> Hi Mark,
>>>>
>>>> Thanks for responding. My brief scan of the literature suggested that
>>>> there are some methods out there to approximate the null space using SVD
>>>> methods, but I wasn't sure how mature these methods were, or if PETSc had
>>>> some capability in this regard.
>>>>
>>>
>>> SVDs will do it but they are expensive.
>>>
>>>
>>>>
>>>> My problem is to wrap a Newton solver around an existing incompressible
>>>> Navier Stokes solver in order to iterate over the NS solution so as to
>>>> determine periodic structures within a weakly nonlinear flow.
>>>>
>>>
>>> Other team members have more experience with INS. We have a lot of
>>> capabilities for this.
>>>
>>> You could start by looking at:
>>>
>>>
>>> https://www.mcs.anl.gov/petsc/documentation/tutorials/HandsOnExercise.html#example4
>>>
>>>
>>>
>>>> I think there is a null space within my problem due to the
>>>> incompressibility constraint (ie: the linear dependence of one velocity
>>>> component on the others). I have some other ideas on how this may be
>>>> removed before assembling the residual, so hopefully I can fix this at the
>>>> level of the residual vector assembly before the problem manifests at the
>>>> level of the Krylov vectors.
>>>>
>>>> Thanks again for clarifying.
>>>>
>>>> Cheers, Dave.
>>>>
>>>> On Thu, Apr 4, 2019 at 8:26 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> The Krylov space can not see the null space (by definition) and so
>>>>> getting a useful near null space from it is not likely.
>>>>>
>>>>> Getting a null space is a hard problem and bootstrap AMG methods, for
>>>>> instance, are developed to try to do that. This is an advanced research
>>>>> topic.
>>>>>
>>>>> You really want to know your null space, what kind of equations do you
>>>>> have?
>>>>>
>>>>> Mark
>>>>>
>>>>> On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hello PETSc,
>>>>>>
>>>>>> I'm attempting to solve a JFNK problem for a system where I only have
>>>>>> a function to compute the residual, but no matrix.
>>>>>>
>>>>>> I wanted to know if there exists functionality in PETSc to do the
>>>>>> following:
>>>>>>
>>>>>> 1) approximate a null space from a set of Krylov vectors
>>>>>>
>>>>>> 2) remove such a null space if it exists
>>>>>>
>>>>>> I'm vaguely familiar with the MatNullSpaceCreate/Remove()
>>>>>> functionality, however I don't know the precise form of a null space, so I
>>>>>> don't have a set of vectors I can assemble and pass to this.
>>>>>>
>>>>>> Cheers, Dave.
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/7dea7f91/attachment.html>

From knepley at gmail.com  Thu Apr  4 07:34:58 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Apr 2019 08:34:58 -0400
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
Message-ID: <CAMYG4GmbViCS4T11_W7oaYFvbiY3s4oBOwFVfDvPyNbNf_cckQ@mail.gmail.com>

On Thu, Apr 4, 2019 at 7:36 AM Dave Lee via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thanks Mark,
>
> I already have the Navier Stokes solver. My issue is wrapping it in a JFNK
> solver to find the periodic solutions. I will keep reading up on SVD
> approaches, there may be some capability for something like this in SLEPc.
>

Dave, why do you have a null space in NS?

  Thanks,

    Matt


> Cheers, Dave.
>
> On Thu, Apr 4, 2019 at 10:32 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> [keep on list]
>>
>> On Thu, Apr 4, 2019 at 7:08 AM Dave Lee <davelee2804 at gmail.com> wrote:
>>
>>> Hi Mark,
>>>
>>> Thanks for responding. My brief scan of the literature suggested that
>>> there are some methods out there to approximate the null space using SVD
>>> methods, but I wasn't sure how mature these methods were, or if PETSc had
>>> some capability in this regard.
>>>
>>
>> SVDs will do it but they are expensive.
>>
>>
>>>
>>> My problem is to wrap a Newton solver around an existing incompressible
>>> Navier Stokes solver in order to iterate over the NS solution so as to
>>> determine periodic structures within a weakly nonlinear flow.
>>>
>>
>> Other team members have more experience with INS. We have a lot of
>> capabilities for this.
>>
>> You could start by looking at:
>>
>>
>> https://www.mcs.anl.gov/petsc/documentation/tutorials/HandsOnExercise.html#example4
>>
>>
>>
>>> I think there is a null space within my problem due to the
>>> incompressibility constraint (ie: the linear dependence of one velocity
>>> component on the others). I have some other ideas on how this may be
>>> removed before assembling the residual, so hopefully I can fix this at the
>>> level of the residual vector assembly before the problem manifests at the
>>> level of the Krylov vectors.
>>>
>>> Thanks again for clarifying.
>>>
>>> Cheers, Dave.
>>>
>>> On Thu, Apr 4, 2019 at 8:26 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> The Krylov space can not see the null space (by definition) and so
>>>> getting a useful near null space from it is not likely.
>>>>
>>>> Getting a null space is a hard problem and bootstrap AMG methods, for
>>>> instance, are developed to try to do that. This is an advanced research
>>>> topic.
>>>>
>>>> You really want to know your null space, what kind of equations do you
>>>> have?
>>>>
>>>> Mark
>>>>
>>>> On Thu, Apr 4, 2019 at 4:28 AM Dave Lee via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hello PETSc,
>>>>>
>>>>> I'm attempting to solve a JFNK problem for a system where I only have
>>>>> a function to compute the residual, but no matrix.
>>>>>
>>>>> I wanted to know if there exists functionality in PETSc to do the
>>>>> following:
>>>>>
>>>>> 1) approximate a null space from a set of Krylov vectors
>>>>>
>>>>> 2) remove such a null space if it exists
>>>>>
>>>>> I'm vaguely familiar with the MatNullSpaceCreate/Remove()
>>>>> functionality, however I don't know the precise form of a null space, so I
>>>>> don't have a set of vectors I can assemble and pass to this.
>>>>>
>>>>> Cheers, Dave.
>>>>>
>>>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/2a9b4ac5/attachment-0001.html>

From jed at jedbrown.org  Thu Apr  4 08:12:05 2019
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 04 Apr 2019 07:12:05 -0600
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
	<CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
Message-ID: <877ec9die2.fsf@jedbrown.org>

Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:

> On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:
>
>> I already have the Navier Stokes solver. My issue is wrapping it in a JFNK
>> solver to find the periodic solutions. I will keep reading up on SVD
>> approaches, there may be some capability for something like this in SLEPc.
>>
>
> Yes, SLEPc will give you parallel eigen solvers, etc.

Even so, computing a null space will be *much* more expensive than solving linear systems.

From bastian.loehrer at tu-dresden.de  Thu Apr  4 09:20:36 2019
From: bastian.loehrer at tu-dresden.de (=?UTF-8?Q?Bastian_L=c3=b6hrer?=)
Date: Thu, 4 Apr 2019 16:20:36 +0200
Subject: [petsc-users] Issue when passing DMDA array on to Paraview
 Catalyst
In-Reply-To: <5BC156E1-D295-411B-AD32-788D57F1C7F5@anl.gov>
References: <136c8d5d-bb94-c5c0-2380-8d4f3a84be00@tu-dresden.de>
	<CAMYG4Gk67ETG1dZU9v_STqYsutQZwfaY507Qdu0W9U+MH-X0Gg@mail.gmail.com>
	<5BC156E1-D295-411B-AD32-788D57F1C7F5@anl.gov>
Message-ID: <77afc235-bb14-dd10-8114-775ad7e9c3e8@tu-dresden.de>

Dear Matt and Barry,

thank you for your responses; they were helpful.

The issue turned out to not be related to PETSc.
The problem was that I tried to pass only a (center) piece of the array 
to Catalyst which is not possible, and which leads to memory issues.

Thanks,
Bastian


On 14.03.19 06:05, Smith, Barry F. wrote:
>> On Mar 13, 2019, at 11:28 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> On Wed, Mar 13, 2019 at 12:16 PM Bastian L?hrer via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> Dear PETSc users,
>>
>> I am having difficulties passing PETSc data on to Paraview Catalyst and
>> it may be related to the way we handle the PETSs data in our Fortran code.
>>
>> We have DMDA objects, which we pass on to subroutines this way:
>>
>>>    ...
>>>    call DMCreateLocalVector(da1dof, loc_p, ierr)
>>>    ...
>>>    call VecGetArray(loc_p, loc_p_v, loc_p_i, ierr)
>>>    call process( loc_p_v(loc_p_i+1) )
>>>    ...
>>>
>> Inside the subroutine (process in this example) we treat the
>> subroutine's argument as if it were an ordinary Fortran array:
>>
>>>    subroutine process( p )
>>>
>>>      use gridinfo ! provides gis, gie, ... etc.
>>>
>>>      implicit none
>>>
>>> #include "petsc_include.h"
>>>
>>>      PetscScalar, dimension(gis:gie,gjs:gje,gks:gke) :: p
>>>      PetscInt i,j,k
>>>
>>>      do k = gks, gke
>>>        do j = gjs, gje
>>>          do i = gis, gie
>>>
>>>              p(i,j,k) = ...
>>>
>>>          enddo
>>>        enddo
>>>      enddo
>>>
>>>    end subroutine process
>>>
>> I find this procedure a little quirky, but it has been working
>> flawlessly for years.
>>
>> However, I am now encountering difficulties when passing this
>> variable/array p on to a Paraview Catalyst adaptor subroutine. Doing so
>> I end up with very strange values there. When replacing p with an
>> ordinary local Fortran array everything is fine.
>>
>> I can't think of a reason it would not work. I would look at the pointer you get inside
>> the Catalyst function using the debugger.
>>
>> Note that you can also get an F90 array out if that is what Catalyst needs.
>    VecGetArrayF90() or DMDAVecGetArrayF90()
>>    Thanks,
>>
>>       Matt
>>   
>> Bastian
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/

-- 
Dipl.-Ing. Bastian L?hrer
Institute of Fluid Mechanics
TU Dresden
Zeuner-Bau Zi 151
George-B?hr-Stra?e 3c
01062 Dresden
tel: 	+49 351 463-32545
fax: 	+49 351 463-35246
mail:	bastian.loehrer at tu-dresden.de


From fdkong.jd at gmail.com  Thu Apr  4 10:14:45 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Thu, 4 Apr 2019 09:14:45 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CA+MQGp8cG68WRJwXQSi03oRe3_rFDZ+nm7MGEc93HAa1RcWugw@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
	<C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
	<CA+MQGp8cG68WRJwXQSi03oRe3_rFDZ+nm7MGEc93HAa1RcWugw@mail.gmail.com>
Message-ID: <CAN5Wd-L6pNXjPffzF__h+EjwJw0Sy1Hb=GBVs=5V_Tj4eoa2ZQ@mail.gmail.com>

Hi Jed,

One more question. Is it fine to use the same SF to exchange two groups of
data at the same time? What is the better way to do this

Fande Kong,

 ierr =
PetscSFReduceBegin(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
 ierr =
PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);
 Doing some calculations
 ierr =
PetscSFReduceEnd(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
 ierr =
PetscSFReduceEnd(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);



On Wed, Apr 3, 2019 at 9:40 PM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

>
>
> On Wed, Apr 3, 2019, 10:29 PM Fande Kong <fdkong.jd at gmail.com> wrote:
>
>> Thanks for the reply.  It is not necessary for me to use MPI_SUM.  I
>> think the better choice is MPIU_REPLACE. Doesn?t MPIU_REPLACE work for any
>> mpi_datatype?
>>
> Yes.
>
>> Fande
>>
>>
>> On Apr 3, 2019, at 9:15 PM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
>>
>>
>> On Wed, Apr 3, 2019 at 3:41 AM Lisandro Dalcin via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the
>>> signedness of plain `char` is implementation (or platform?) dependent.
>>>  I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you
>>> should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that
>>> MPI_SIGNED_CHAR is from MPI 2.0.
>>>
>>
>> MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER
>> (which represent printable characters) cannot be used in reduction
>> operations"
>> So Fande's code and Jed's branch have problems. To fix that, we have to
>> add support for signed char, unsigned char, and char in PetscSF.  The first
>> two types support add, mult, logical and bitwise operations. The last is a
>> dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would
>> raise error on Fande's code. I can come up with a fix tomorrow.
>>
>>
>>>
>>> On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi All,
>>>>
>>>> There were some error messages when using PetscSFReduceBegin with
>>>> MPI_CHAR.
>>>>
>>>> ierr =
>>>> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>>>>
>>>>
>>>> My question would be: Does PetscSFReduceBegin suppose work with
>>>> MPI_CHAR? If not, should we document somewhere?
>>>>
>>>> Thanks
>>>>
>>>> Fande,
>>>>
>>>>
>>>> [*0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------*
>>>> *[0]PETSC ERROR: No support for this operation for this object type*
>>>> *[0]PETSC ERROR: No support for type size not divisible by 4*
>>>> *[0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
>>>> *[0]PETSC ERROR: Petsc Development GIT revision:
>>>> v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600*
>>>> *[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------*
>>>> *[1]PETSC ERROR: No support for this operation for this object type*
>>>> *[1]PETSC ERROR: No support for type size not divisible by 4*
>>>> *[1]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
>>>> *[1]PETSC ERROR: Petsc Development GIT revision:
>>>> v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600*
>>>> *[1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once
>>>> named fn605731.local by kongf Tue Apr  2 21:48:41 2019*
>>>> *[1]PETSC ERROR: Configure options --download-hypre=1
>>>> --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1
>>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1
>>>> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
>>>> --download-party --download-chaco --with-cxx-dialect=C++11*
>>>> *[1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *[1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *[1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named
>>>> fn605731.local by kongf Tue Apr  2 21:48:41 2019*
>>>> *[0]PETSC ERROR: Configure options --download-hypre=1
>>>> --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1
>>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1
>>>> PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch
>>>> --download-party --download-chaco --with-cxx-dialect=C++11*
>>>> *[0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *[0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *[0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c*
>>>> *[0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
>>>> *[0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850
>>>> in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>>>> *[0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
>>>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>>>> *[0]PETSC ERROR: #7 MatPtAP() line 9429 in
>>>> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
>>>> *[0]PETSC ERROR: #8 main() line 58 in
>>>> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
>>>> *[0]PETSC ERROR: PETSc Option Table entries:*
>>>> *[0]PETSC ERROR: -matptap_via allatonce*
>>>> *[0]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov
>>>> <petsc-maint at mcs.anl.gov>----------*
>>>> *[1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in
>>>> /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c*
>>>> *[1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850
>>>> in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>>>> *[1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in
>>>> /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c*
>>>> *[1]PETSC ERROR: #7 MatPtAP() line 9429 in
>>>> /Users/kongf/projects/petsc/src/mat/interface/matrix.c*
>>>> *[1]PETSC ERROR: #8 main() line 58 in
>>>> /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c*
>>>> *[1]PETSC ERROR: PETSc Option Table entries:*
>>>> *[1]PETSC ERROR: -matptap_via allatonce*
>>>> *[1]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov
>>>> <petsc-maint at mcs.anl.gov>----------*
>>>>
>>>> *--------------------------------------------------------------------------*
>>>> *MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD*
>>>> *with errorcode 56.*
>>>>
>>>> *NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.*
>>>> *You may or may not see output from other processes, depending on*
>>>> *exactly when Open MPI kills them.*
>>>>
>>>> *--------------------------------------------------------------------------*
>>>> *[fn605731.local:78133] 1 more process has sent help message
>>>> help-mpi-api.txt / mpi-abort*
>>>> *[fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to
>>>> 0 to see all help / error messages*
>>>>
>>>
>>>
>>> --
>>> Lisandro Dalcin
>>> ============
>>> Research Scientist
>>> Extreme Computing Research Center (ECRC)
>>> King Abdullah University of Science and Technology (KAUST)
>>> http://ecrc.kaust.edu.sa/
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/73fea621/attachment.html>

From jed at jedbrown.org  Thu Apr  4 10:30:27 2019
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 04 Apr 2019 09:30:27 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
Message-ID: <87tvfdbxf0.fsf@jedbrown.org>

"Zhang, Junchao via petsc-users" <petsc-users at mcs.anl.gov> writes:

> MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations"
> So Fande's code and Jed's branch have problems. To fix that, we have to add support for signed char, unsigned char, and char in PetscSF.  The first two types support add, mult, logical and bitwise operations. The last is a dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would raise error on Fande's code. I can come up with a fix tomorrow.

FWIW, my patch did not define arithmetic or comparison operations.  It
only added moving bytes for sizes smaller than MPI_INT.  Perhaps it
should, but that would require identifying more types, as you say.

From zakaryah at gmail.com  Thu Apr  4 10:35:35 2019
From: zakaryah at gmail.com (zakaryah)
Date: Thu, 4 Apr 2019 11:35:35 -0400
Subject: [petsc-users] HDF5 VecView segfaults with more than 1 processor
Message-ID: <CABm8tB7HnaPzj8=GjwncisBC6rOR6Vj3K_GJ+GeqDhW-S8sWVw@mail.gmail.com>

I'm trying to output PETSc vectors from a DMComposite using VecView with an
HDF5 viewer.  This works fine with -n 1, but segfaults in parallel.  The
code is simple:

ierr=PetscViewerHDF5Open(PETSC_COMM_WORLD,fname,FILE_MODE_WRITE,&H5viewer);CHKERRQ(ierr);
ierr=VecView(X,H5viewer);CHKERRQ(ierr);

ierr=PetscViewerDestroy(&H5viewer);CHKERRQ(ierr);


I'm using PETSc 3.11.0, configured with --with-hdf5
--with-hdf5-include=/MYFS/hdf5-1.10.5/include/
--with-hdf5-lib=/MYFS/hdf5-1.10.5/lib/libhdf5.so, where hdf5-1.10.5 was
configured with mpicc, and --enable-parallel, per the instructions in the
INSTALL_parallel doc.  I'm using openmpi 1.10.3a1.

According to valgrind, the segfault originates in H5G_traverse, trying to
write 8 bytes to a null pointer (n -3):

==35379== Thread 1:



==35379== Invalid write of size 8



==35379==    at 0x1AEBD92B: ADIOI_Flatten (flatten.c:225)



==35379==    by 0x1AEBF82E: ADIOI_Flatten_datatype (flatten.c:80)



==35379==    by 0x1AEB5982: ADIO_Set_view (ad_set_view.c:52)



==35379==    by 0x1AE9B860: mca_io_romio_dist_MPI_File_set_view
(set_view.c:155)


==35379==    by 0x92C0D3D: PMPI_File_set_view (pfile_set_view.c:75)



==35379==    by 0x830D963: H5FD_mpio_write (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x8101576: H5FD_write (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80DD30B: H5F__accum_write (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x81F3830: H5PB_write (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80E906A: H5F_block_write (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x808FEBB: H5D__chunk_allocate (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80A1774: H5D__init_storage (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80A7282: H5D__alloc_storage (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80AE4A4: H5D__layout_oh_create (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80A3EF2: H5D__create (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80AF768: H5O__dset_create (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x81A3898: H5O_obj_create (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x81675E3: H5L__link_cb (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x8137680: H5G__traverse_real (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x8137E93: H5G_traverse (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

There are also many "possibly lost" warnings, starting in
PetscViewerHDF5Open but arising from libhdf5, for example:

==35379== 10 bytes in 1 blocks are possibly lost in loss record 3,259 of
14,406


==35379==    at 0x4C29BFD: malloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)



==35379==    by 0x810D495: H5FL_blk_malloc (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x81F8F10: H5RS_create (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x8129F3B: H5G__name_init (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x81318AF: H5G_mkroot (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80E7016: H5F_open (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x80D7C62: H5Fcreate (in
/rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)



==35379==    by 0x565F43C: PetscViewerFileSetName_HDF5 (hdf5v.c:314)



==35379==    by 0x568A828: PetscViewerFileSetName (filev.c:667)



==35379==    by 0x56624DD: PetscViewerHDF5Open (hdf5v.c:547)



==35379==    by 0x434092: myFunction (myFunction.c:276)



==35379==    by 0x42DC03: main (main.c:280)

With n -1, I don't get any of these errors and there is no segfault.

I suppose this is an HDF5 problem but I'm wondering if anyone has advice
for me on how to fix this.  If there are issues with parallel HDF5 then I'm
happy to settle for sequential VecView, but I couldn't figure out how to
get that to work either.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/e481f112/attachment-0001.html>

From jed at jedbrown.org  Thu Apr  4 10:36:56 2019
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 04 Apr 2019 09:36:56 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <CAN5Wd-L6pNXjPffzF__h+EjwJw0Sy1Hb=GBVs=5V_Tj4eoa2ZQ@mail.gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
	<C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
	<CA+MQGp8cG68WRJwXQSi03oRe3_rFDZ+nm7MGEc93HAa1RcWugw@mail.gmail.com>
	<CAN5Wd-L6pNXjPffzF__h+EjwJw0Sy1Hb=GBVs=5V_Tj4eoa2ZQ@mail.gmail.com>
Message-ID: <87sguxbx47.fsf@jedbrown.org>

Fande Kong via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi Jed,
>
> One more question. Is it fine to use the same SF to exchange two groups of
> data at the same time? What is the better way to do this

This should work due to the non-overtaking property defined by MPI.

> Fande Kong,
>
>  ierr =
> PetscSFReduceBegin(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
>  ierr =
> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);
>  Doing some calculations
>  ierr =
> PetscSFReduceEnd(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
>  ierr =
> PetscSFReduceEnd(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);

From pghysels at lbl.gov  Thu Apr  4 13:45:22 2019
From: pghysels at lbl.gov (Pieter Ghysels)
Date: Thu, 4 Apr 2019 11:45:22 -0700
Subject: [petsc-users] max_it for EPS minres preconditioner
Message-ID: <CAG23jbThVXnikBQe__N+3JPCtrwveVU3O+zE9T-SAwn7+Cu0+w@mail.gmail.com>

Hi,

I'm trying to set the maximum number of iterations in a minres
preconditioner for the lobpcg eigensolver from SLEPc.
Using KSPSetTolerances, I can change the minres tolerance, but not maxit
(it's always 5).

  ierr = EPSCreate( PETSC_COMM_WORLD , &eps ) ;              CHKERRQ( ierr
) ;
  ...

  ST st;
  KSP ksp;
  PC pc;
  ierr = EPSGetST( eps , &st ) ;                             CHKERRQ( ierr
) ;
  ierr = STSetType( st, STPRECOND ) ;                        CHKERRQ( ierr
) ;
  ierr = STGetKSP( st , &ksp ) ;                             CHKERRQ( ierr
) ;
  ierr = KSPSetType( ksp , KSPMINRES ) ;                     CHKERRQ( ierr
) ;
  ierr = KSPGetPC( ksp , &pc ) ;                             CHKERRQ( ierr
) ;
  ierr = PCSetType( pc , PCNONE ) ;                          CHKERRQ( ierr
) ;
*  ierr = KSPSetTolerances*
*    ( ksp , /*tol_prec*/ 1e-10 , PETSC_DEFAULT , PETSC_DEFAULT ,
/*maxit_prec*/ 7 ) ;*
  CHKERRQ( ierr ) ;
  ierr = KSPSetFromOptions( ksp ) ;                          CHKERRQ( ierr
) ;
  ierr = STSetFromOptions( st ) ;                            CHKERRQ( ierr
) ;
  ierr = EPSSetFromOptions( eps ) ;                          CHKERRQ( ierr
) ;

  ...
 ierr = EPSSetOperators( eps , mat , NULL ) ;             CHKERRQ( ierr ) ;

 ierr = EPSSolve( eps ) ;                                 CHKERRQ( ierr ) ;



When I run with -eps_view, I see:

...
EPS Object: 4 MPI processes
  type: lobpcg
...
ST Object: 4 MPI processes
  type: precond
  shift: 0.
  number of matrices: 1
  KSP Object: (st_) 4 MPI processes
    type: minres
*    maximum iterations=5, initial guess is zero*
*    tolerances:  relative=1e-10, absolute=1e-50, divergence=10000.*
    left preconditioning
    using PRECONDITIONED norm type for convergence test
  PC Object: (st_) 4 MPI processes
    type: none
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/60facacc/attachment.html>

From fdkong.jd at gmail.com  Thu Apr  4 13:53:36 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Thu, 4 Apr 2019 12:53:36 -0600
Subject: [petsc-users] How to use khash in PETSc?
In-Reply-To: <CAEcYPwAiXwE80fM6AG_iybiK-zwyRHEqDwybQEZ_TDiB8WdOag@mail.gmail.com>
References: <CAN5Wd-KCApHobhN33-F5TabrWZKbVUku_=Y=RZU7oyhNjPmF6g@mail.gmail.com>
	<CAMYG4G=tXk8PFkeBOizWzeu+C1cM8YaMuMCeM-J0hKe6X7EaTg@mail.gmail.com>
	<CAEcYPwAiXwE80fM6AG_iybiK-zwyRHEqDwybQEZ_TDiB8WdOag@mail.gmail.com>
Message-ID: <CAN5Wd-LwaCR=UPi=MG0wrWCbNN32WD1KQh5eyLzKw=SZ5xvtYg@mail.gmail.com>

Thanks Lisandro,

PetscHSetI and PetscHMapI work pretty well.

Even though I did not do any performance profiling, I have some questions
on this.

What is the right way to preallocate the memory for PetscHSetI and
PetscHMapI? Just call "XXXResize()"?  Should we always need to figure out
the right size of the hash table? Or just leave the hash table
automatically reallocate the memory?   Dynamic memory reallocation will
hurt any performance?

Thanks,

Fande,



On Mon, Mar 25, 2019 at 3:37 AM Lisandro Dalcin <dalcinl at gmail.com> wrote:

>
>
> On Mon, 25 Mar 2019 at 02:40, Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> On Sun, Mar 24, 2019 at 12:38 PM Fande Kong via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi All,
>>>
>>> Since PetscTable will be replaced by khash in the future somehow,  it is
>>> better to use khash for new implementations. I was wondering where I can
>>> find some examples that use khash? Do we have any petsc wrappers of khash?
>>>
>>
>> First look here:
>>
>>
>> https://bitbucket.org/petsc/petsc/src/master/include/petsc/private/hashseti.h
>>
>>
>
>> and the other associated files (setij, mapi, mapij). Lisandro did a good
>> job organizing
>> these, and we might have already defined what you want.
>>
>>
> It is even documented, which is extremely unusual coming from me!!!
>
> --
> Lisandro Dalcin
> ============
> Research Scientist
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/7c96fa1c/attachment.html>

From knepley at gmail.com  Thu Apr  4 14:12:03 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Apr 2019 15:12:03 -0400
Subject: [petsc-users] How to use khash in PETSc?
In-Reply-To: <CAN5Wd-LwaCR=UPi=MG0wrWCbNN32WD1KQh5eyLzKw=SZ5xvtYg@mail.gmail.com>
References: <CAN5Wd-KCApHobhN33-F5TabrWZKbVUku_=Y=RZU7oyhNjPmF6g@mail.gmail.com>
	<CAMYG4G=tXk8PFkeBOizWzeu+C1cM8YaMuMCeM-J0hKe6X7EaTg@mail.gmail.com>
	<CAEcYPwAiXwE80fM6AG_iybiK-zwyRHEqDwybQEZ_TDiB8WdOag@mail.gmail.com>
	<CAN5Wd-LwaCR=UPi=MG0wrWCbNN32WD1KQh5eyLzKw=SZ5xvtYg@mail.gmail.com>
Message-ID: <CAMYG4GnveN9bMBz51KR1z2KZ1VZUVF8JVSSd5X7BUs4Nk1LGRA@mail.gmail.com>

On Thu, Apr 4, 2019 at 2:53 PM Fande Kong <fdkong.jd at gmail.com> wrote:

> Thanks Lisandro,
>
> PetscHSetI and PetscHMapI work pretty well.
>
> Even though I did not do any performance profiling, I have some questions
> on this.
>
> What is the right way to preallocate the memory for PetscHSetI and
> PetscHMapI? Just call "XXXResize()"?  Should we always need to figure out
> the right size of the hash table? Or just leave the hash table
> automatically reallocate the memory?   Dynamic memory reallocation will
> hurt any performance?
>

Unless you know the approximate size, its fine to have it dynamically
allocate.

  Matt


> Thanks,
>
> Fande,
>
>
>
> On Mon, Mar 25, 2019 at 3:37 AM Lisandro Dalcin <dalcinl at gmail.com> wrote:
>
>>
>>
>> On Mon, 25 Mar 2019 at 02:40, Matthew Knepley via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> On Sun, Mar 24, 2019 at 12:38 PM Fande Kong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Since PetscTable will be replaced by khash in the future somehow,  it
>>>> is better to use khash for new implementations. I was wondering where I can
>>>> find some examples that use khash? Do we have any petsc wrappers of khash?
>>>>
>>>
>>> First look here:
>>>
>>>
>>> https://bitbucket.org/petsc/petsc/src/master/include/petsc/private/hashseti.h
>>>
>>>
>>
>>> and the other associated files (setij, mapi, mapij). Lisandro did a good
>>> job organizing
>>> these, and we might have already defined what you want.
>>>
>>>
>> It is even documented, which is extremely unusual coming from me!!!
>>
>> --
>> Lisandro Dalcin
>> ============
>> Research Scientist
>> Extreme Computing Research Center (ECRC)
>> King Abdullah University of Science and Technology (KAUST)
>> http://ecrc.kaust.edu.sa/
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/11de7216/attachment.html>

From jroman at dsic.upv.es  Thu Apr  4 14:29:53 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 4 Apr 2019 21:29:53 +0200
Subject: [petsc-users] max_it for EPS minres preconditioner
In-Reply-To: <CAG23jbThVXnikBQe__N+3JPCtrwveVU3O+zE9T-SAwn7+Cu0+w@mail.gmail.com>
References: <CAG23jbThVXnikBQe__N+3JPCtrwveVU3O+zE9T-SAwn7+Cu0+w@mail.gmail.com>
Message-ID: <95F760DC-7EAC-484A-BB1F-DD687D47C958@dsic.upv.es>

Oops, the intention was to set 5 as the default value, but it is set at EPSSetUp() so changing it before is useless. This is wrong behaviour. I will try to fix it tomorrow.
Jose


> El 4 abr 2019, a las 20:45, Pieter Ghysels via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hi,
> 
> I'm trying to set the maximum number of iterations in a minres preconditioner for the lobpcg eigensolver from SLEPc.
> Using KSPSetTolerances, I can change the minres tolerance, but not maxit (it's always 5).
> 
>   ierr = EPSCreate( PETSC_COMM_WORLD , &eps ) ;              CHKERRQ( ierr ) ;
>   ...
> 
>   ST st;
>   KSP ksp;
>   PC pc;
>   ierr = EPSGetST( eps , &st ) ;                             CHKERRQ( ierr ) ;
>   ierr = STSetType( st, STPRECOND ) ;                        CHKERRQ( ierr ) ;
>   ierr = STGetKSP( st , &ksp ) ;                             CHKERRQ( ierr ) ;
>   ierr = KSPSetType( ksp , KSPMINRES ) ;                     CHKERRQ( ierr ) ;
>   ierr = KSPGetPC( ksp , &pc ) ;                             CHKERRQ( ierr ) ;
>   ierr = PCSetType( pc , PCNONE ) ;                          CHKERRQ( ierr ) ;
>   ierr = KSPSetTolerances
>     ( ksp , /*tol_prec*/ 1e-10 , PETSC_DEFAULT , PETSC_DEFAULT , /*maxit_prec*/ 7 ) ;
>   CHKERRQ( ierr ) ;
>   ierr = KSPSetFromOptions( ksp ) ;                          CHKERRQ( ierr ) ;
>   ierr = STSetFromOptions( st ) ;                            CHKERRQ( ierr ) ;
>   ierr = EPSSetFromOptions( eps ) ;                          CHKERRQ( ierr ) ;
> 
>   ...
>  ierr = EPSSetOperators( eps , mat , NULL ) ;             CHKERRQ( ierr ) ;
> 
>  ierr = EPSSolve( eps ) ;                                 CHKERRQ( ierr ) ;
> 
> 
> 
> When I run with -eps_view, I see:
> 
> ...
> EPS Object: 4 MPI processes
>   type: lobpcg
> ...
> ST Object: 4 MPI processes
>   type: precond
>   shift: 0.
>   number of matrices: 1
>   KSP Object: (st_) 4 MPI processes
>     type: minres
>     maximum iterations=5, initial guess is zero
>     tolerances:  relative=1e-10, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using PRECONDITIONED norm type for convergence test
>   PC Object: (st_) 4 MPI processes
>     type: none
> ...
> 


From yyang85 at stanford.edu  Thu Apr  4 15:15:58 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 4 Apr 2019 20:15:58 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
Message-ID: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I'm trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I'm getting the below error:

[1]PETSC ERROR: Invalid argument
[1]PETSC ERROR: Insufficient data, read only 0 < 1 items
[1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c

Is there something wrong with how I'm implementing this, or ASCIIRead does not work with multiple processors?

Thanks,
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/00fdc8be/attachment-0001.html>

From jacoboest87 at gmail.com  Thu Apr  4 15:43:12 2019
From: jacoboest87 at gmail.com (Jacob Oest)
Date: Thu, 4 Apr 2019 22:43:12 +0200
Subject: [petsc-users] PCMG Solver Issue
Message-ID: <CABK210_jQZjZ5b=a8syPaoRk-vGpUk92SJMUAL4mzLvE1tKC3A@mail.gmail.com>

Dear all,

I am using petsc4py and can't seem to figure out an issue.
I need to solve the linear system KU = P, and K and U are correctly set up,
and using default ksp solvers i can easily solve the system. However, I am
trying to solve with pcmg and have two issues:
a) I get 'PETSC Error code 73 - object is in wrong state' when i try to
solve or setup the ksp.
b) What is the petsc4py equivalent to PCMGSetGalerkin? Can't seem to figure
it out.

Below is how I set up the solver, next is the error message, and lastly is
the result from KSPView()

Thanks in advance for any help you can provide!

*THE CODE:*
        RTOL = 1.0e-5
        ATOL = 1.0e-50
        DTOL = 1.0e3
        MAXITSGLOBAL = 200
        COARSE_RTOL = 1.0e-8
        COARSE_ATOL = 1.0e-50
        COARSE_DTOL = 1e3
        COARSE_MAXITS = 30
        COARSE_RESTART = 30
        SMOOTH_SWEEPS = 4

        # Fine grid Krylov Method
        self.ksp = petsc.KSP().create()
        self.ksp.setType('fgmres')
        self.ksp.setTolerances(rtol=RTOL, atol=ATOL, divtol=DTOL,
max_it=MAXITSGLOBAL)
        self.ksp.setInitialGuessNonzero(True)

        # Set operators
        self.ksp.setOperators(mesh.K, mesh.K)

        # Get Preconditioner and set it up
        pc = self.ksp.getPC()
        pc.setType('mg')

        self.ksp.setFromOptions()

        # Set all levels in the PCMG (zeroth entry is finest level):
        da_list = list()
        da_list.append(mesh.da_nodes)
        da_list.extend([da_obj for da_obj in
mesh.da_nodes.coarsenHierarchy(3)])

        # PCMG specific options
        pc.setMGLevels(mesh.mg_levels)
        pc.setMGType(0)  # multiplicative
        pc.setMGCycleType(1)  # v

        for k in range(1, mesh.mg_levels):
            r, _ = da_list[k].createInterpolation(da_list[k-1])
            pc.setMGInterpolation(k, r)

        # Avoid the default for the mg part
        cksp = pc.getMGCoarseSolve()
        cksp.setType('gmres')
        cksp.setGMRESRestart(COARSE_RESTART)
        cksp.setTolerances(rtol=COARSE_RTOL, atol=COARSE_ATOL,
divtol=COARSE_DTOL, max_it=COARSE_MAXITS)
        cpc = cksp.getPC()
        cpc.setType('sor')

        # Set smoothers on all levels
        for k in range(1, mesh.mg_levels):
            dksp = pc.getMGSmoother(k)
            dpc = dksp.getPC()
            dksp.setType('gmres')
            dksp.setGMRESRestart(SMOOTH_SWEEPS)
            dksp.setTolerances(max_it=SMOOTH_SWEEPS)
            dpc.setType('sor')


*The Error:*

petsc4py.PETSc.Error: error code 73

[0] KSPSolve() line 725 in
/tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/itfunc.c

[0] KSPSetUp() line 391 in
/tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/itfunc.c

[0] PCSetUp() line 932 in
/tmp/pip-req-build-goqa79r0/src/ksp/pc/interface/precon.c

[0] PCSetUp_MG() line 819 in
/tmp/pip-req-build-goqa79r0/src/ksp/pc/impls/mg/mg.c

[0] KSPCreateVecs() line 947 in
/tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/iterativ.c

[0] Object is in wrong state

[0] You requested a vector from a KSP that cannot provide one


*The KSPView():*

KSP Object: 1 MPI processes

  type: fgmres

    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement

    happy breakdown tolerance 1e-30

  maximum iterations=200, nonzero initial guess

  tolerances:  relative=1e-05, absolute=1e-50, divergence=1000.

  left preconditioning

  using DEFAULT norm type for convergence test

PC Object: 1 MPI processes

  type: mg

  PC has not been set up so information may be incomplete

    type is MULTIPLICATIVE, levels=4 cycles=v

      Cycles per PCApply=1

      Not using Galerkin computed coarse grid matrices

  Coarse grid solver -- level -------------------------------

    KSP Object: (mg_coarse_) 1 MPI processes

      type: gmres

        restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement

        happy breakdown tolerance 1e-30

      maximum iterations=30, initial guess is zero

      tolerances:  relative=1e-08, absolute=1e-50, divergence=1000.

      left preconditioning

      using DEFAULT norm type for convergence test

    PC Object: (mg_coarse_) 1 MPI processes

      type: sor

      PC has not been set up so information may be incomplete

        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.

  Down solver (pre-smoother) on level 1 -------------------------------

    KSP Object: (mg_levels_1_) 1 MPI processes

      type: gmres

        restart=4, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement

        happy breakdown tolerance 1e-30

      maximum iterations=4, initial guess is zero

      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.

      left preconditioning

      using NONE norm type for convergence test

    PC Object: (mg_levels_1_) 1 MPI processes

      type: sor

      PC has not been set up so information may be incomplete

        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.

  Up solver (post-smoother) same as down solver (pre-smoother)

  Down solver (pre-smoother) on level 2 -------------------------------

    KSP Object: (mg_levels_2_) 1 MPI processes

      type: gmres

        restart=4, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement

        happy breakdown tolerance 1e-30

      maximum iterations=4, initial guess is zero

      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.

      left preconditioning

      using NONE norm type for convergence test

    PC Object: (mg_levels_2_) 1 MPI processes

      type: sor

      PC has not been set up so information may be incomplete

        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.

  Up solver (post-smoother) same as down solver (pre-smoother)

  Down solver (pre-smoother) on level 3 -------------------------------

    KSP Object: (mg_levels_3_) 1 MPI processes

      type: gmres

        restart=4, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement

        happy breakdown tolerance 1e-30

      maximum iterations=4, initial guess is zero

      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.

      left preconditioning

      using NONE norm type for convergence test

    PC Object: (mg_levels_3_) 1 MPI processes

      type: sor

      PC has not been set up so information may be incomplete

        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.

  Up solver (post-smoother) same as down solver (pre-smoother)

  linear system matrix = precond matrix:

  Mat Object: 1 MPI processes

    type: seqaij

    rows=4131, cols=4131, bs=3

    total: nonzeros=275625, allocated nonzeros=275625

    total number of mallocs used during MatSetValues calls =0

      using I-node routines: found 1377 nodes, limit used is 5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/86a74ed0/attachment.html>

From pghysels at lbl.gov  Thu Apr  4 16:37:14 2019
From: pghysels at lbl.gov (Pieter Ghysels)
Date: Thu, 4 Apr 2019 14:37:14 -0700
Subject: [petsc-users] max_it for EPS minres preconditioner
In-Reply-To: <95F760DC-7EAC-484A-BB1F-DD687D47C958@dsic.upv.es>
References: <CAG23jbThVXnikBQe__N+3JPCtrwveVU3O+zE9T-SAwn7+Cu0+w@mail.gmail.com>
	<95F760DC-7EAC-484A-BB1F-DD687D47C958@dsic.upv.es>
Message-ID: <CAG23jbRSqSkfMGLNVa3J8S=uaD=WgyW-Zz2qZsvLu5eC2-p=Hg@mail.gmail.com>

Dear Jose,

It indeed works correctly when I set maxit after calling EPSSetUp.
Thanks!

Pieter

On Thu, Apr 4, 2019 at 12:30 PM Jose E. Roman <jroman at dsic.upv.es> wrote:

> Oops, the intention was to set 5 as the default value, but it is set at
> EPSSetUp() so changing it before is useless. This is wrong behaviour. I
> will try to fix it tomorrow.
> Jose
>
>
> > El 4 abr 2019, a las 20:45, Pieter Ghysels via petsc-users <
> petsc-users at mcs.anl.gov> escribi?:
> >
> > Hi,
> >
> > I'm trying to set the maximum number of iterations in a minres
> preconditioner for the lobpcg eigensolver from SLEPc.
> > Using KSPSetTolerances, I can change the minres tolerance, but not maxit
> (it's always 5).
> >
> >   ierr = EPSCreate( PETSC_COMM_WORLD , &eps ) ;              CHKERRQ(
> ierr ) ;
> >   ...
> >
> >   ST st;
> >   KSP ksp;
> >   PC pc;
> >   ierr = EPSGetST( eps , &st ) ;                             CHKERRQ(
> ierr ) ;
> >   ierr = STSetType( st, STPRECOND ) ;                        CHKERRQ(
> ierr ) ;
> >   ierr = STGetKSP( st , &ksp ) ;                             CHKERRQ(
> ierr ) ;
> >   ierr = KSPSetType( ksp , KSPMINRES ) ;                     CHKERRQ(
> ierr ) ;
> >   ierr = KSPGetPC( ksp , &pc ) ;                             CHKERRQ(
> ierr ) ;
> >   ierr = PCSetType( pc , PCNONE ) ;                          CHKERRQ(
> ierr ) ;
> >   ierr = KSPSetTolerances
> >     ( ksp , /*tol_prec*/ 1e-10 , PETSC_DEFAULT , PETSC_DEFAULT ,
> /*maxit_prec*/ 7 ) ;
> >   CHKERRQ( ierr ) ;
> >   ierr = KSPSetFromOptions( ksp ) ;                          CHKERRQ(
> ierr ) ;
> >   ierr = STSetFromOptions( st ) ;                            CHKERRQ(
> ierr ) ;
> >   ierr = EPSSetFromOptions( eps ) ;                          CHKERRQ(
> ierr ) ;
> >
> >   ...
> >  ierr = EPSSetOperators( eps , mat , NULL ) ;             CHKERRQ( ierr
> ) ;
> >
> >  ierr = EPSSolve( eps ) ;                                 CHKERRQ( ierr
> ) ;
> >
> >
> >
> > When I run with -eps_view, I see:
> >
> > ...
> > EPS Object: 4 MPI processes
> >   type: lobpcg
> > ...
> > ST Object: 4 MPI processes
> >   type: precond
> >   shift: 0.
> >   number of matrices: 1
> >   KSP Object: (st_) 4 MPI processes
> >     type: minres
> >     maximum iterations=5, initial guess is zero
> >     tolerances:  relative=1e-10, absolute=1e-50, divergence=10000.
> >     left preconditioning
> >     using PRECONDITIONED norm type for convergence test
> >   PC Object: (st_) 4 MPI processes
> >     type: none
> > ...
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/a64ae68c/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Apr  4 16:51:59 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 4 Apr 2019 21:51:59 +0000
Subject: [petsc-users] PCMG Solver Issue
In-Reply-To: <CABK210_jQZjZ5b=a8syPaoRk-vGpUk92SJMUAL4mzLvE1tKC3A@mail.gmail.com>
References: <CABK210_jQZjZ5b=a8syPaoRk-vGpUk92SJMUAL4mzLvE1tKC3A@mail.gmail.com>
Message-ID: <1E57DE7C-2A3D-49EB-9858-62B67913157B@anl.gov>


   Jacob,

       The two errors are related. PCMG uses KSPCreateVecs() on each level to create work vectors and KSPCreateVecs() uses the attached matrix to create the work vectors (via MatCreateVecs()). Since the Mat on the intermediate levels has not been created yet the error is generated while attempting to create the vectors. It you were able to set the PCMGSetGalerkin() option then the intermediate matrices would exist and the code would not fail at creating work vectors.

    I git grep the petsc4py code and it looks like inadvertently PCMGSetGalerkin() is missing. Someone who knows petsc4py will have to add that interface.

    For now you can try setting the option -pc_mg_galerkin both before calling KSPSetFromOptions.

   Barry




> On Apr 4, 2019, at 3:43 PM, Jacob Oest via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear all,
> 
> I am using petsc4py and can't seem to figure out an issue.
> I need to solve the linear system KU = P, and K and U are correctly set up, and using default ksp solvers i can easily solve the system. However, I am trying to solve with pcmg and have two issues:
> a) I get 'PETSC Error code 73 - object is in wrong state' when i try to solve or setup the ksp.
> b) What is the petsc4py equivalent to PCMGSetGalerkin? Can't seem to figure it out.
> 
> Below is how I set up the solver, next is the error message, and lastly is the result from KSPView()
> 
> Thanks in advance for any help you can provide!
> 
> THE CODE:
>         RTOL = 1.0e-5
>         ATOL = 1.0e-50
>         DTOL = 1.0e3
>         MAXITSGLOBAL = 200
>         COARSE_RTOL = 1.0e-8
>         COARSE_ATOL = 1.0e-50
>         COARSE_DTOL = 1e3
>         COARSE_MAXITS = 30
>         COARSE_RESTART = 30
>         SMOOTH_SWEEPS = 4
> 
>         # Fine grid Krylov Method
>         self.ksp = petsc.KSP().create()
>         self.ksp.setType('fgmres')
>         self.ksp.setTolerances(rtol=RTOL, atol=ATOL, divtol=DTOL, max_it=MAXITSGLOBAL)
>         self.ksp.setInitialGuessNonzero(True)
> 
>         # Set operators
>         self.ksp.setOperators(mesh.K, mesh.K)
> 
>         # Get Preconditioner and set it up
>         pc = self.ksp.getPC()
>         pc.setType('mg')
> 
>         self.ksp.setFromOptions()
> 
>         # Set all levels in the PCMG (zeroth entry is finest level):
>         da_list = list()
>         da_list.append(mesh.da_nodes)
>         da_list.extend([da_obj for da_obj in mesh.da_nodes.coarsenHierarchy(3)])
> 
>         # PCMG specific options
>         pc.setMGLevels(mesh.mg_levels)
>         pc.setMGType(0)  # multiplicative
>         pc.setMGCycleType(1)  # v
> 
>         for k in range(1, mesh.mg_levels):
>             r, _ = da_list[k].createInterpolation(da_list[k-1])
>             pc.setMGInterpolation(k, r)
> 
>         # Avoid the default for the mg part
>         cksp = pc.getMGCoarseSolve()
>         cksp.setType('gmres')
>         cksp.setGMRESRestart(COARSE_RESTART)
>         cksp.setTolerances(rtol=COARSE_RTOL, atol=COARSE_ATOL, divtol=COARSE_DTOL, max_it=COARSE_MAXITS)
>         cpc = cksp.getPC()
>         cpc.setType('sor')
> 
>         # Set smoothers on all levels
>         for k in range(1, mesh.mg_levels):
>             dksp = pc.getMGSmoother(k)
>             dpc = dksp.getPC()
>             dksp.setType('gmres')
>             dksp.setGMRESRestart(SMOOTH_SWEEPS)
>             dksp.setTolerances(max_it=SMOOTH_SWEEPS)
>             dpc.setType('sor')
> 
> 
> The Error:
> petsc4py.PETSc.Error: error code 73
> [0] KSPSolve() line 725 in /tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/itfunc.c
> [0] KSPSetUp() line 391 in /tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/itfunc.c
> [0] PCSetUp() line 932 in /tmp/pip-req-build-goqa79r0/src/ksp/pc/interface/precon.c
> [0] PCSetUp_MG() line 819 in /tmp/pip-req-build-goqa79r0/src/ksp/pc/impls/mg/mg.c
> [0] KSPCreateVecs() line 947 in /tmp/pip-req-build-goqa79r0/src/ksp/ksp/interface/iterativ.c
> [0] Object is in wrong state
> [0] You requested a vector from a KSP that cannot provide one
> 
> The KSPView():
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=200, nonzero initial guess
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1000.
>   left preconditioning
>   using DEFAULT norm type for convergence test
> PC Object: 1 MPI processes
>   type: mg
>   PC has not been set up so information may be incomplete
>     type is MULTIPLICATIVE, levels=4 cycles=v
>       Cycles per PCApply=1
>       Not using Galerkin computed coarse grid matrices
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: gmres
>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=30, initial guess is zero
>       tolerances:  relative=1e-08, absolute=1e-50, divergence=1000.
>       left preconditioning
>       using DEFAULT norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>       PC has not been set up so information may be incomplete
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: gmres
>         restart=4, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=4, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>       PC has not been set up so information may be incomplete
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: gmres
>         restart=4, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=4, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>       PC has not been set up so information may be incomplete
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: gmres
>         restart=4, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=4, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>       PC has not been set up so information may be incomplete
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=4131, cols=4131, bs=3
>     total: nonzeros=275625, allocated nonzeros=275625
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 1377 nodes, limit used is 5
> 


From bsmith at mcs.anl.gov  Thu Apr  4 17:07:37 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 4 Apr 2019 22:07:37 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>


   Currently PetscViewerFileSetName_ASCII() only opens the file on process 0 (for read or write) thus when you call PetscViewerASCIIRead() from any process but the first it will be reading from an fd that has not been set and you could get unpredictable results. 

   The implementation and documentation for PetscViewerASCIIRead() is buggy.

   There are two possible fixes we could make

1) have PetscViewerFileSetName_ASCII()  open the file for reading on all processes or
2) have PetscViewerASCIIRead() generate an error if the process is not rank == 0

   Barry

Note that using PetscViewerASCIIRead() from a handful of processes is probably fine but having hundreds or thousands of processes open the same ASCII file and reading from it will likely not be scalable.




> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello team,
>  
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>  
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>  
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>  
> Thanks,
> Yuyun


From jczhang at mcs.anl.gov  Thu Apr  4 17:34:57 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 4 Apr 2019 22:34:57 +0000
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <87sguxbx47.fsf@jedbrown.org>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
	<C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
	<CA+MQGp8cG68WRJwXQSi03oRe3_rFDZ+nm7MGEc93HAa1RcWugw@mail.gmail.com>
	<CAN5Wd-L6pNXjPffzF__h+EjwJw0Sy1Hb=GBVs=5V_Tj4eoa2ZQ@mail.gmail.com>
	<87sguxbx47.fsf@jedbrown.org>
Message-ID: <CA+MQGp9H5A1kmZQz4p5bjP0uAFhPrthi+8vKM-3iwrewm_834g@mail.gmail.com>

I updated the branch and made a PR. I tried to do MPI_SUM on MPI_CHAR. We do not have UnpackAdd on this type (we are right). But unfortunately, MPICH's MPI_Reduce_local did not report errors (it should) so we did not generate an error either.

--Junchao Zhang


On Thu, Apr 4, 2019 at 10:37 AM Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>> wrote:
Fande Kong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> writes:

> Hi Jed,
>
> One more question. Is it fine to use the same SF to exchange two groups of
> data at the same time? What is the better way to do this

This should work due to the non-overtaking property defined by MPI.

> Fande Kong,
>
>  ierr =
> PetscSFReduceBegin(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
>  ierr =
> PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);
>  Doing some calculations
>  ierr =
> PetscSFReduceEnd(ptap->sf,MPIU_INT,rmtspace,space,MPIU_REPLACE);CHKERRQ(ierr);
>  ierr =
> PetscSFReduceEnd(ptap->sf,MPI_CHAR,rmtspace2,space2,MPIU_REPLACE);CHKERRQ(ierr);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/2a7b246d/attachment.html>

From davelee2804 at gmail.com  Thu Apr  4 20:11:21 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Fri, 5 Apr 2019 12:11:21 +1100
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <877ec9die2.fsf@jedbrown.org>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
	<CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
	<877ec9die2.fsf@jedbrown.org>
Message-ID: <CAC4fGOHLpM15FP1=xzY-MPgY3bM53iHAgqyF+L1XC6egKxj3nQ@mail.gmail.com>

Hey Matt,

I'm not solving NS per se, but rather wrapping up a Navier Stokes solver
within a SNES to iterate over the solution of the Navier Stokes equations
with a certain time period in order to determine approximate periodic
solutions to the NS equations (with some corrections).

My residual vector is basically a difference between the final and initial
velocity states of the NS solve (with corrections). However since one
component can be diagnosed from the others via the divergence free
condition (up to a constant), I suspect that maybe what I should be doing
is just omit one of the velocity components from the residual vector, and
then diagnose this from the others via incompressibility, rather than try
and correct for this after the vectors have already been assembled. This is
all outside the scope of my PETSc question, and I don't expect you to have
an answer, just mentioning it since you asked.

Cheers, Dave.

On Fri, Apr 5, 2019 at 12:12 AM Jed Brown <jed at jedbrown.org> wrote:

> Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:
> >
> >> I already have the Navier Stokes solver. My issue is wrapping it in a
> JFNK
> >> solver to find the periodic solutions. I will keep reading up on SVD
> >> approaches, there may be some capability for something like this in
> SLEPc.
> >>
> >
> > Yes, SLEPc will give you parallel eigen solvers, etc.
>
> Even so, computing a null space will be *much* more expensive than solving
> linear systems.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/cf1d6391/attachment.html>

From yyang85 at stanford.edu  Thu Apr  4 20:19:01 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 5 Apr 2019 01:19:01 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
Message-ID: <DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>

We are probably not going to use hundreds of processors, but i think it would be good to just have processor 0 read the input and broadcast that to all the other processors. Would that be a possible fix? And what would you suggest to work around this problem for now?

Thanks!
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Thursday, April 4, 2019 3:07:37 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] ASCIIRead error for multiple processors


   Currently PetscViewerFileSetName_ASCII() only opens the file on process 0 (for read or write) thus when you call PetscViewerASCIIRead() from any process but the first it will be reading from an fd that has not been set and you could get unpredictable results.

   The implementation and documentation for PetscViewerASCIIRead() is buggy.

   There are two possible fixes we could make

1) have PetscViewerFileSetName_ASCII()  open the file for reading on all processes or
2) have PetscViewerASCIIRead() generate an error if the process is not rank == 0

   Barry

Note that using PetscViewerASCIIRead() from a handful of processes is probably fine but having hundreds or thousands of processes open the same ASCII file and reading from it will likely not be scalable.




> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Hello team,
>
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>
> Thanks,
> Yuyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/e00733c5/attachment-0001.html>

From knepley at gmail.com  Thu Apr  4 21:26:52 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Apr 2019 22:26:52 -0400
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAC4fGOHLpM15FP1=xzY-MPgY3bM53iHAgqyF+L1XC6egKxj3nQ@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
	<CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
	<877ec9die2.fsf@jedbrown.org>
	<CAC4fGOHLpM15FP1=xzY-MPgY3bM53iHAgqyF+L1XC6egKxj3nQ@mail.gmail.com>
Message-ID: <CAMYG4Gnw3vcHQFLPD2YHaN=6Bh3UWnOo6K3UTJsxE4=4gRg3KA@mail.gmail.com>

On Thu, Apr 4, 2019 at 9:11 PM Dave Lee <davelee2804 at gmail.com> wrote:

> Hey Matt,
>
> I'm not solving NS per se, but rather wrapping up a Navier Stokes solver
> within a SNES to iterate over the solution of the Navier Stokes equations
> with a certain time period in order to determine approximate periodic
> solutions to the NS equations (with some corrections).
>
> My residual vector is basically a difference between the final and initial
> velocity states of the NS solve (with corrections). However since one
> component can be diagnosed from the others via the divergence free
> condition (up to a constant), I suspect that maybe what I should be doing
> is just omit one of the velocity components from the residual vector, and
> then diagnose this from the others via incompressibility, rather than try
> and correct for this after the vectors have already been assembled. This is
> all outside the scope of my PETSc question, and I don't expect you to have
> an answer, just mentioning it since you asked.
>

Interesting.  It sounds like you can impose this condition the same way we
impose \int p = 0.

  Thanks,

    Matt


> Cheers, Dave.
>
> On Fri, Apr 5, 2019 at 12:12 AM Jed Brown <jed at jedbrown.org> wrote:
>
>> Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:
>>
>> > On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:
>> >
>> >> I already have the Navier Stokes solver. My issue is wrapping it in a
>> JFNK
>> >> solver to find the periodic solutions. I will keep reading up on SVD
>> >> approaches, there may be some capability for something like this in
>> SLEPc.
>> >>
>> >
>> > Yes, SLEPc will give you parallel eigen solvers, etc.
>>
>> Even so, computing a null space will be *much* more expensive than
>> solving linear systems.
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/531364b4/attachment.html>

From knepley at gmail.com  Thu Apr  4 21:30:20 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Apr 2019 22:30:20 -0400
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>

On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> We are probably not going to use hundreds of processors, but i think it
> would be good to just have processor 0 read the input and broadcast that to
> all the other processors. Would that be a possible fix? And what would you
> suggest to work around this problem for now?
>

Explicitly call MPI_Bcast().

   Matt


> Thanks!
> Yuyun
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> ------------------------------
> *From:* Smith, Barry F. <bsmith at mcs.anl.gov>
> *Sent:* Thursday, April 4, 2019 3:07:37 PM
> *To:* Yuyun Yang
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>
>
>    Currently PetscViewerFileSetName_ASCII() only opens the file on process
> 0 (for read or write) thus when you call PetscViewerASCIIRead() from any
> process but the first it will be reading from an fd that has not been set
> and you could get unpredictable results.
>
>    The implementation and documentation for PetscViewerASCIIRead() is
> buggy.
>
>    There are two possible fixes we could make
>
> 1) have PetscViewerFileSetName_ASCII()  open the file for reading on all
> processes or
> 2) have PetscViewerASCIIRead() generate an error if the process is not
> rank == 0
>
>    Barry
>
> Note that using PetscViewerASCIIRead() from a handful of processes is
> probably fine but having hundreds or thousands of processes open the same
> ASCII file and reading from it will likely not be scalable.
>
>
>
>
> > On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello team,
> >
> > I?m trying to use PetscViewerASCIIRead() to read in a single
> interger/scalar value from an input file. It works for one processor.
> However, when running on multiple processors, I?m getting the below error:
> >
> > [1]PETSC ERROR: Invalid argument
> > [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> > [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in
> /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
> >
> > Is there something wrong with how I?m implementing this, or ASCIIRead
> does not work with multiple processors?
> >
> > Thanks,
> > Yuyun
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190404/2e9af731/attachment.html>

From yyang85 at stanford.edu  Thu Apr  4 21:56:47 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 5 Apr 2019 02:56:47 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
Message-ID: <DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>

So do I call MPI_Bcast right after I call PetscViewerASCIIRead? Is that going to prevent the other processors from trying to read the same file but were unable to?

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, April 4, 2019 7:30:20 PM
To: Yuyun Yang
Cc: Smith, Barry F.; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
We are probably not going to use hundreds of processors, but i think it would be good to just have processor 0 read the input and broadcast that to all the other processors. Would that be a possible fix? And what would you suggest to work around this problem for now?

Explicitly call MPI_Bcast().

   Matt

Thanks!
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>
Sent: Thursday, April 4, 2019 3:07:37 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors


   Currently PetscViewerFileSetName_ASCII() only opens the file on process 0 (for read or write) thus when you call PetscViewerASCIIRead() from any process but the first it will be reading from an fd that has not been set and you could get unpredictable results.

   The implementation and documentation for PetscViewerASCIIRead() is buggy.

   There are two possible fixes we could make

1) have PetscViewerFileSetName_ASCII()  open the file for reading on all processes or
2) have PetscViewerASCIIRead() generate an error if the process is not rank == 0

   Barry

Note that using PetscViewerASCIIRead() from a handful of processes is probably fine but having hundreds or thousands of processes open the same ASCII file and reading from it will likely not be scalable.




> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hello team,
>
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>
> Thanks,
> Yuyun



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/b4fcdb21/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr  4 23:58:13 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 5 Apr 2019 04:58:13 +0000
Subject: [petsc-users] HDF5 VecView segfaults with more than 1 processor
In-Reply-To: <CABm8tB7HnaPzj8=GjwncisBC6rOR6Vj3K_GJ+GeqDhW-S8sWVw@mail.gmail.com>
References: <CABm8tB7HnaPzj8=GjwncisBC6rOR6Vj3K_GJ+GeqDhW-S8sWVw@mail.gmail.com>
Message-ID: <9A3D926A-E23A-4E54-A858-44FA4ADC1F6D@anl.gov>


   Would you be able to send us a sample code that reproduces the problem? We don't have a lot of tests for DMComposite so it is possible there are bugs in our VecView() for that case.

   Barry

   Also the "possibly lost" messages from valgrind are irrelevant after a crash or valgrind errors so can be ignored in that situation.


> On Apr 4, 2019, at 10:35 AM, zakaryah via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> I'm trying to output PETSc vectors from a DMComposite using VecView with an HDF5 viewer.  This works fine with -n 1, but segfaults in parallel.  The code is simple:
> 
> ierr=PetscViewerHDF5Open(PETSC_COMM_WORLD,fname,FILE_MODE_WRITE,&H5viewer);CHKERRQ(ierr);
> ierr=VecView(X,H5viewer);CHKERRQ(ierr);
> ierr=PetscViewerDestroy(&H5viewer);CHKERRQ(ierr);
> 
> I'm using PETSc 3.11.0, configured with --with-hdf5 --with-hdf5-include=/MYFS/hdf5-1.10.5/include/ --with-hdf5-lib=/MYFS/hdf5-1.10.5/lib/libhdf5.so, where hdf5-1.10.5 was configured with mpicc, and --enable-parallel, per the instructions in the INSTALL_parallel doc.  I'm using openmpi 1.10.3a1.
> 
> According to valgrind, the segfault originates in H5G_traverse, trying to write 8 bytes to a null pointer (n -3):
> 
> ==35379== Thread 1:                                                                                                                                                                                             
> ==35379== Invalid write of size 8                                                                                                                                                                               
> ==35379==    at 0x1AEBD92B: ADIOI_Flatten (flatten.c:225)                                                                                                                                                       
> ==35379==    by 0x1AEBF82E: ADIOI_Flatten_datatype (flatten.c:80)                                                                                                                                               
> ==35379==    by 0x1AEB5982: ADIO_Set_view (ad_set_view.c:52)                                                                                                                                                    
> ==35379==    by 0x1AE9B860: mca_io_romio_dist_MPI_File_set_view (set_view.c:155)                                                                                                                                
> ==35379==    by 0x92C0D3D: PMPI_File_set_view (pfile_set_view.c:75)                                                                                                                                             
> ==35379==    by 0x830D963: H5FD_mpio_write (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                     
> ==35379==    by 0x8101576: H5FD_write (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                          
> ==35379==    by 0x80DD30B: H5F__accum_write (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                    
> ==35379==    by 0x81F3830: H5PB_write (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                          
> ==35379==    by 0x80E906A: H5F_block_write (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                     
> ==35379==    by 0x808FEBB: H5D__chunk_allocate (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                 
> ==35379==    by 0x80A1774: H5D__init_storage (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                   
> ==35379==    by 0x80A7282: H5D__alloc_storage (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                  
> ==35379==    by 0x80AE4A4: H5D__layout_oh_create (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                               
> ==35379==    by 0x80A3EF2: H5D__create (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                         
> ==35379==    by 0x80AF768: H5O__dset_create (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                    
> ==35379==    by 0x81A3898: H5O_obj_create (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                      
> ==35379==    by 0x81675E3: H5L__link_cb (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                        
> ==35379==    by 0x8137680: H5G__traverse_real (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                  
> ==35379==    by 0x8137E93: H5G_traverse (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                        
> ==35379==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> 
> There are also many "possibly lost" warnings, starting in PetscViewerHDF5Open but arising from libhdf5, for example:
> 
> ==35379== 10 bytes in 1 blocks are possibly lost in loss record 3,259 of 14,406                                                                                                                                 
> ==35379==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)                                                                                                                    
> ==35379==    by 0x810D495: H5FL_blk_malloc (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                     
> ==35379==    by 0x81F8F10: H5RS_create (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                         
> ==35379==    by 0x8129F3B: H5G__name_init (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                      
> ==35379==    by 0x81318AF: H5G_mkroot (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                          
> ==35379==    by 0x80E7016: H5F_open (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                            
> ==35379==    by 0x80D7C62: H5Fcreate (in /rugpfs/fs0/home/zfrentz/hdf5-1.10.5/lib/libhdf5.so.103.1.0)                                                                                                           
> ==35379==    by 0x565F43C: PetscViewerFileSetName_HDF5 (hdf5v.c:314)                                                                                                                                            
> ==35379==    by 0x568A828: PetscViewerFileSetName (filev.c:667)                                                                                                                                                 
> ==35379==    by 0x56624DD: PetscViewerHDF5Open (hdf5v.c:547)                                                                                                                                                    
> ==35379==    by 0x434092: myFunction (myFunction.c:276)                                                                                                                                                 
> ==35379==    by 0x42DC03: main (main.c:280)
> 
> With n -1, I don't get any of these errors and there is no segfault.
> 
> I suppose this is an HDF5 problem but I'm wondering if anyone has advice for me on how to fix this.  If there are issues with parallel HDF5 then I'm happy to settle for sequential VecView, but I couldn't figure out how to get that to work either.
> 


From davelee2804 at gmail.com  Fri Apr  5 00:57:56 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Fri, 5 Apr 2019 16:57:56 +1100
Subject: [petsc-users] testing for and removing a null space using JFNK
In-Reply-To: <CAMYG4Gnw3vcHQFLPD2YHaN=6Bh3UWnOo6K3UTJsxE4=4gRg3KA@mail.gmail.com>
References: <CAC4fGOF7Nbhos3BHbLhr1zaJbix35CP+rY4kKzSXrnGF0Gm-rw@mail.gmail.com>
	<CADOhEh53iyVQOAAoh-O_ePCo3s3WVOJaZC1dA-kZAZAOrHAJZA@mail.gmail.com>
	<CAC4fGOEuC=Z534MHOs9bV16u+OhCjajEuWM26d36_tyOM92i2w@mail.gmail.com>
	<CADOhEh7piAd7phcLK=5_D1zsPJqqWNfo6+skKQLtX-VV=X_nzA@mail.gmail.com>
	<CAC4fGOGFbBvVX0Ct5mW4QbBQSYdB81qMy5qVjXXKvJXpKVrurg@mail.gmail.com>
	<CADOhEh58uAYzCoAeEhUczvAOBwG56dQ9MTuHcgS04R55KtLTjA@mail.gmail.com>
	<877ec9die2.fsf@jedbrown.org>
	<CAC4fGOHLpM15FP1=xzY-MPgY3bM53iHAgqyF+L1XC6egKxj3nQ@mail.gmail.com>
	<CAMYG4Gnw3vcHQFLPD2YHaN=6Bh3UWnOo6K3UTJsxE4=4gRg3KA@mail.gmail.com>
Message-ID: <CAC4fGOH3yyHnEdF_Eu9gQTXst-J_VQnGOYSB+5RdBE8cDz07_A@mail.gmail.com>

Hi Matt,

Perhaps. I need to think more about weather this system is over-determined,
and if so how to recover the velocity components from the Newton iteration.

Thanks for your help.

Cheers, Dave.

On Fri, Apr 5, 2019 at 1:27 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Apr 4, 2019 at 9:11 PM Dave Lee <davelee2804 at gmail.com> wrote:
>
>> Hey Matt,
>>
>> I'm not solving NS per se, but rather wrapping up a Navier Stokes solver
>> within a SNES to iterate over the solution of the Navier Stokes equations
>> with a certain time period in order to determine approximate periodic
>> solutions to the NS equations (with some corrections).
>>
>> My residual vector is basically a difference between the final and
>> initial velocity states of the NS solve (with corrections). However since
>> one component can be diagnosed from the others via the divergence free
>> condition (up to a constant), I suspect that maybe what I should be doing
>> is just omit one of the velocity components from the residual vector, and
>> then diagnose this from the others via incompressibility, rather than try
>> and correct for this after the vectors have already been assembled. This is
>> all outside the scope of my PETSc question, and I don't expect you to have
>> an answer, just mentioning it since you asked.
>>
>
> Interesting.  It sounds like you can impose this condition the same way we
> impose \int p = 0.
>
>   Thanks,
>
>     Matt
>
>
>> Cheers, Dave.
>>
>> On Fri, Apr 5, 2019 at 12:12 AM Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:
>>>
>>> > On Thu, Apr 4, 2019 at 7:35 AM Dave Lee <davelee2804 at gmail.com> wrote:
>>> >
>>> >> I already have the Navier Stokes solver. My issue is wrapping it in a
>>> JFNK
>>> >> solver to find the periodic solutions. I will keep reading up on SVD
>>> >> approaches, there may be some capability for something like this in
>>> SLEPc.
>>> >>
>>> >
>>> > Yes, SLEPc will give you parallel eigen solvers, etc.
>>>
>>> Even so, computing a null space will be *much* more expensive than
>>> solving linear systems.
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/315f8d2c/attachment.html>

From eda.oktay at metu.edu.tr  Fri Apr  5 02:20:15 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Fri, 5 Apr 2019 10:20:15 +0300
Subject: [petsc-users] error: Petsc has generated inconsistent data,
 MPI_Allreduce() called in different locations (code lines) on
 different processors
Message-ID: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>

Hello,

I am trying to calculate unweighted Laplacian of a matrix by using 2 cores.
If the size of matrix is in even number then my program works. However,
when I try to use a matrix having odd number for size, I guess since size
of the matrix cannot be divided into processors correctly, I get the
following error:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Petsc has generated inconsistent data
[1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
on different processors
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
09:50:54 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
09:50:54 2019
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
#1 MatStashScatterBegin_BTS() line 843 in
/home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
[1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
#2 MatStashScatterBegin_BTS() line 843 in
/home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
[1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
/home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
[0]PETSC ERROR: #3 main() line 164 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
[1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
/home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -f
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
#5 MatAssemblyBegin() line 5251 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
[1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
[0]PETSC ERROR: -unweighted
#6 main() line 169 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message
-------send entire error message to petsc-maint at mcs.anl.gov----------
-f
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
[1]PETSC ERROR: -mat_partitioning_type parmetis
[1]PETSC ERROR: -unweighted
[1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
----------------End of Error Message -------send entire error message to
petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1


 where line 164 in my main program is MatSetOption and line 169
is MatAssemblyBegin. I am new to MPI usage so I did not understand why
MPI_Allreduce() causes a problem and how can I fix the problem.

Thank you,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/375f5012/attachment.html>

From jroman at dsic.upv.es  Fri Apr  5 04:36:34 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 5 Apr 2019 11:36:34 +0200
Subject: [petsc-users] max_it for EPS minres preconditioner
In-Reply-To: <CAG23jbRSqSkfMGLNVa3J8S=uaD=WgyW-Zz2qZsvLu5eC2-p=Hg@mail.gmail.com>
References: <CAG23jbThVXnikBQe__N+3JPCtrwveVU3O+zE9T-SAwn7+Cu0+w@mail.gmail.com>
	<95F760DC-7EAC-484A-BB1F-DD687D47C958@dsic.upv.es>
	<CAG23jbRSqSkfMGLNVa3J8S=uaD=WgyW-Zz2qZsvLu5eC2-p=Hg@mail.gmail.com>
Message-ID: <32129053-7FC1-485D-B747-1CC1E0C741F7@dsic.upv.es>

I have made a fix in branch jose/maint/change-maxit-lobpcg
It will be included in maint once the nightly tests are clean.
Thanks for reporting this.
Jose

> El 4 abr 2019, a las 23:37, Pieter Ghysels <pghysels at lbl.gov> escribi?:
> 
> Dear Jose,
> 
> It indeed works correctly when I set maxit after calling EPSSetUp.
> Thanks!
> 
> Pieter
> 
> On Thu, Apr 4, 2019 at 12:30 PM Jose E. Roman <jroman at dsic.upv.es> wrote:
> Oops, the intention was to set 5 as the default value, but it is set at EPSSetUp() so changing it before is useless. This is wrong behaviour. I will try to fix it tomorrow.
> Jose
> 
> 
> > El 4 abr 2019, a las 20:45, Pieter Ghysels via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> > 
> > Hi,
> > 
> > I'm trying to set the maximum number of iterations in a minres preconditioner for the lobpcg eigensolver from SLEPc.
> > Using KSPSetTolerances, I can change the minres tolerance, but not maxit (it's always 5).
> > 
> >   ierr = EPSCreate( PETSC_COMM_WORLD , &eps ) ;              CHKERRQ( ierr ) ;
> >   ...
> > 
> >   ST st;
> >   KSP ksp;
> >   PC pc;
> >   ierr = EPSGetST( eps , &st ) ;                             CHKERRQ( ierr ) ;
> >   ierr = STSetType( st, STPRECOND ) ;                        CHKERRQ( ierr ) ;
> >   ierr = STGetKSP( st , &ksp ) ;                             CHKERRQ( ierr ) ;
> >   ierr = KSPSetType( ksp , KSPMINRES ) ;                     CHKERRQ( ierr ) ;
> >   ierr = KSPGetPC( ksp , &pc ) ;                             CHKERRQ( ierr ) ;
> >   ierr = PCSetType( pc , PCNONE ) ;                          CHKERRQ( ierr ) ;
> >   ierr = KSPSetTolerances
> >     ( ksp , /*tol_prec*/ 1e-10 , PETSC_DEFAULT , PETSC_DEFAULT , /*maxit_prec*/ 7 ) ;
> >   CHKERRQ( ierr ) ;
> >   ierr = KSPSetFromOptions( ksp ) ;                          CHKERRQ( ierr ) ;
> >   ierr = STSetFromOptions( st ) ;                            CHKERRQ( ierr ) ;
> >   ierr = EPSSetFromOptions( eps ) ;                          CHKERRQ( ierr ) ;
> > 
> >   ...
> >  ierr = EPSSetOperators( eps , mat , NULL ) ;             CHKERRQ( ierr ) ;
> > 
> >  ierr = EPSSolve( eps ) ;                                 CHKERRQ( ierr ) ;
> > 
> > 
> > 
> > When I run with -eps_view, I see:
> > 
> > ...
> > EPS Object: 4 MPI processes
> >   type: lobpcg
> > ...
> > ST Object: 4 MPI processes
> >   type: precond
> >   shift: 0.
> >   number of matrices: 1
> >   KSP Object: (st_) 4 MPI processes
> >     type: minres
> >     maximum iterations=5, initial guess is zero
> >     tolerances:  relative=1e-10, absolute=1e-50, divergence=10000.
> >     left preconditioning
> >     using PRECONDITIONED norm type for convergence test
> >   PC Object: (st_) 4 MPI processes
> >     type: none
> > ...
> > 
> 


From diogo.ferreira-sabino at univ-tlse3.fr  Fri Apr  5 06:19:34 2019
From: diogo.ferreira-sabino at univ-tlse3.fr (Diogo FERREIRA SABINO)
Date: Fri, 5 Apr 2019 13:19:34 +0200 (CEST)
Subject: [petsc-users] Constructing a MATNEST with blocks defined in
 different procs
Message-ID: <429247112.120532.1554463174712@scout-f5.univ-toulouse.fr>

Hi,
I'm new in petsc and I'm trying to construct a MATNEST in two procs, by setting each block of the nested matrix with a MATMPIAIJ matrix defined in each proc.
I'm trying to use MatCreateNest or MatNestSetSubMats, but I'm not being able to do it.
Using MatNestSetSubMats, I'm trying to construct the MATNEST, giving a pointer to the correct matrices depending on the MPIrank of that proc.
I'm obtaining the error message for the line :MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Wrong type of object: Parameter # 5

Is there a way of doing it, or all the blocks of the MATNEST have to exist in the same communicator as the MATNEST matrix?
A simple test is given below, lunching it with: mpirun -n 2 ./Main_petsc.exe

static char help[]     = "Create MPI Nest";
#include <petscmat.h>

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  PetscInitialize(&argc,&argv,(char*)0,help);
  //////////////////////////////////////////////////////////////////////////////
  PetscErrorCode  ierr;
  PetscMPIInt MPIrank,MPIsize;
  MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
  MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);

  ////////////////////////////////////////////////////////   Create Each Matrix:
  Mat Adiag;

  //Create a Adiag different on each proc:
  ierr = MatCreate(PETSC_COMM_SELF,&Adiag);                     CHKERRQ(ierr);
  ierr = MatSetSizes(Adiag,2,2,PETSC_DECIDE,PETSC_DECIDE);      CHKERRQ(ierr);
  ierr = MatSetType(Adiag,MATMPIAIJ);                           CHKERRQ(ierr);
  ierr = MatSetFromOptions(Adiag);                              CHKERRQ(ierr);
  ierr = MatMPIAIJSetPreallocation(Adiag,2,NULL,2,NULL);        CHKERRQ(ierr);

  MatSetValue(Adiag,0,0,(MPIrank+5),INSERT_VALUES);
  MatSetValue(Adiag,0,1,(MPIrank+10),INSERT_VALUES);
  MatSetValue(Adiag,1,0,(MPIrank+15),INSERT_VALUES);
  MatSetValue(Adiag,1,1,(MPIrank+20),INSERT_VALUES);
  MatAssemblyBegin(Adiag,MAT_FINAL_ASSEMBLY);     MatAssemblyEnd(Adiag,MAT_FINAL_ASSEMBLY);

  ///////////////////////////////////////////////////////////////   Create Nest:
  MPI_Barrier(PETSC_COMM_WORLD);
  Mat       AfullNEST, *AfullNESTpointer;

  PetscMalloc1(2,&AfullNESTpointer);
  AfullNESTpointer[0]=NULL;
  AfullNESTpointer[1]=NULL;
  AfullNESTpointer[MPIrank]=Adiag;
  // Rank=0 --> AfullNESTpointer[0]=Adiag; AfullNESTpointer[1]=NULL;
  // Rank=1 --> AfullNESTpointer[0]=NULL;  AfullNESTpointer[1]=Adiag;

  IS        IS_ROW,IS_COL;
  ISCreateStride(PETSC_COMM_SELF,1,MPIrank,0,&IS_ROW);
  ISCreateStride(PETSC_COMM_SELF,2,0,1,&IS_COL);
  // Rank=0 --> IS_ROW= [ 0 ] ; IS_COL= [ 0, 1 ] ;
  // Rank=1 --> IS_ROW= [ 1 ] ; IS_COL= [ 0, 1 ] ;

  MatCreate(PETSC_COMM_WORLD,&AfullNEST);
  MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
  // MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
  // MatCreateNest(PETSC_COMM_WORLD,1,&IS_ROW,1,&IS_COL,AfullNESTpointer,&AfullNEST);
    ierr = MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer); CHKERRQ(ierr);

  ierr = PetscFinalize(); CHKERRQ(ierr);
  return 0;
}

From knepley at gmail.com  Fri Apr  5 07:16:12 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Apr 2019 08:16:12 -0400
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
	<DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4G=Ax4yrfVA6cWoQ4M2KkqAOmX+thf5T22FaxJtODSM88g@mail.gmail.com>

On Thu, Apr 4, 2019 at 10:56 PM Yuyun Yang <yyang85 at stanford.edu> wrote:

> So do I call MPI_Bcast right after I call PetscViewerASCIIRead? Is that
> going to prevent the other processors from trying to read the same file but
> were unable to?
>

No, all this does is replicate data from process 0 on the other processes.

   Matt


> Thanks,
> Yuyun
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, April 4, 2019 7:30:20 PM
> *To:* Yuyun Yang
> *Cc:* Smith, Barry F.; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>
> On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> We are probably not going to use hundreds of processors, but i think it
>> would be good to just have processor 0 read the input and broadcast that to
>> all the other processors. Would that be a possible fix? And what would you
>> suggest to work around this problem for now?
>>
>
> Explicitly call MPI_Bcast().
>
>    Matt
>
>
>> Thanks!
>> Yuyun
>>
>> Get Outlook for iOS <https://aka.ms/o0ukef>
>> ------------------------------
>> *From:* Smith, Barry F. <bsmith at mcs.anl.gov>
>> *Sent:* Thursday, April 4, 2019 3:07:37 PM
>> *To:* Yuyun Yang
>> *Cc:* petsc-users at mcs.anl.gov
>> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>>
>>
>>    Currently PetscViewerFileSetName_ASCII() only opens the file on
>> process 0 (for read or write) thus when you call PetscViewerASCIIRead()
>> from any process but the first it will be reading from an fd that has not
>> been set and you could get unpredictable results.
>>
>>    The implementation and documentation for PetscViewerASCIIRead() is
>> buggy.
>>
>>    There are two possible fixes we could make
>>
>> 1) have PetscViewerFileSetName_ASCII()  open the file for reading on all
>> processes or
>> 2) have PetscViewerASCIIRead() generate an error if the process is not
>> rank == 0
>>
>>    Barry
>>
>> Note that using PetscViewerASCIIRead() from a handful of processes is
>> probably fine but having hundreds or thousands of processes open the same
>> ASCII file and reading from it will likely not be scalable.
>>
>>
>>
>>
>> > On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Hello team,
>> >
>> > I?m trying to use PetscViewerASCIIRead() to read in a single
>> interger/scalar value from an input file. It works for one processor.
>> However, when running on multiple processors, I?m getting the below error:
>> >
>> > [1]PETSC ERROR: Invalid argument
>> > [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
>> > [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in
>> /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>> >
>> > Is there something wrong with how I?m implementing this, or ASCIIRead
>> does not work with multiple processors?
>> >
>> > Thanks,
>> > Yuyun
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/1855d880/attachment-0001.html>

From knepley at gmail.com  Fri Apr  5 07:20:33 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Apr 2019 08:20:33 -0400
Subject: [petsc-users] error: Petsc has generated inconsistent data,
 MPI_Allreduce() called in different locations (code lines) on
 different processors
In-Reply-To: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>
References: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>
Message-ID: <CAMYG4Gk8_XSH9YSXR4Fmv8EPcRMYhiRHYjyo3F0Tbav6bdF9Gg@mail.gmail.com>

On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I am trying to calculate unweighted Laplacian of a matrix by using 2
> cores. If the size of matrix is in even number then my program works.
> However, when I try to use a matrix having odd number for size, I guess
> since size of the matrix cannot be divided into processors correctly, I get
> the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Petsc has generated inconsistent data
> [1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
> on different processors
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
> on different processors
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #1 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #2 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [0]PETSC ERROR: #3 main() line 164 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> #5 MatAssemblyBegin() line 5251 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> #6 main() line 169 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message
> -------send entire error message to petsc-maint at mcs.anl.gov----------
> -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -unweighted
> [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> ----------------End of Error Message -------send entire error message to
> petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>
>
>  where line 164 in my main program is MatSetOption and line 169
> is MatAssemblyBegin. I am new to MPI usage so I did not understand why
> MPI_Allreduce() causes a problem and how can I fix the problem.
>

You have to call collective methods on all processes in the same order.
This is not happening in your code. Beyond that,
there is no way for us to tell how this happened.

  Thanks,

     Matt


> Thank you,
>
> Eda
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/3e3fd595/attachment.html>

From mfadams at lbl.gov  Fri Apr  5 07:38:46 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 5 Apr 2019 08:38:46 -0400
Subject: [petsc-users] Constructing a MATNEST with blocks defined in
 different procs
In-Reply-To: <429247112.120532.1554463174712@scout-f5.univ-toulouse.fr>
References: <429247112.120532.1554463174712@scout-f5.univ-toulouse.fr>
Message-ID: <CADOhEh5whnESzNaf3JmP-j8DfzVVFDBary=6Ur2e6VtNSA-tiQ@mail.gmail.com>

On Fri, Apr 5, 2019 at 7:19 AM Diogo FERREIRA SABINO via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
> I'm new in petsc and I'm trying to construct a MATNEST in two procs, by
> setting each block of the nested matrix with a MATMPIAIJ matrix defined in
> each proc.
> I'm trying to use MatCreateNest or MatNestSetSubMats, but I'm not being
> able to do it.
> Using MatNestSetSubMats, I'm trying to construct the MATNEST, giving a
> pointer to the correct matrices depending on the MPIrank of that proc.
> I'm obtaining the error message for the line
> :MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Wrong type of object: Parameter # 5
>
> Is there a way of doing it, or all the blocks of the MATNEST have to exist
> in the same communicator as the MATNEST matrix?
>

Yes, they must all have the same communicator. A matrix can be empty on a
process, so you just create them with the global communicator, set the
local sizes that you want (eg, 0 on some procs).


> A simple test is given below, lunching it with: mpirun -n 2
> ./Main_petsc.exe
>
> static char help[]     = "Create MPI Nest";
> #include <petscmat.h>
>
> #undef __FUNCT__
> #define __FUNCT__ "main"
> int main(int argc,char **argv)
> {
>   PetscInitialize(&argc,&argv,(char*)0,help);
>
> //////////////////////////////////////////////////////////////////////////////
>   PetscErrorCode  ierr;
>   PetscMPIInt MPIrank,MPIsize;
>   MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
>   MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);
>
>   ////////////////////////////////////////////////////////   Create Each
> Matrix:
>   Mat Adiag;
>
>   //Create a Adiag different on each proc:
>   ierr = MatCreate(PETSC_COMM_SELF,&Adiag);
>  CHKERRQ(ierr);
>   ierr = MatSetSizes(Adiag,2,2,PETSC_DECIDE,PETSC_DECIDE);
> CHKERRQ(ierr);
>   ierr = MatSetType(Adiag,MATMPIAIJ);
>  CHKERRQ(ierr);
>   ierr = MatSetFromOptions(Adiag);
> CHKERRQ(ierr);
>   ierr = MatMPIAIJSetPreallocation(Adiag,2,NULL,2,NULL);
> CHKERRQ(ierr);
>
>   MatSetValue(Adiag,0,0,(MPIrank+5),INSERT_VALUES);
>   MatSetValue(Adiag,0,1,(MPIrank+10),INSERT_VALUES);
>   MatSetValue(Adiag,1,0,(MPIrank+15),INSERT_VALUES);
>   MatSetValue(Adiag,1,1,(MPIrank+20),INSERT_VALUES);
>   MatAssemblyBegin(Adiag,MAT_FINAL_ASSEMBLY);
>  MatAssemblyEnd(Adiag,MAT_FINAL_ASSEMBLY);
>
>   ///////////////////////////////////////////////////////////////   Create
> Nest:
>   MPI_Barrier(PETSC_COMM_WORLD);
>   Mat       AfullNEST, *AfullNESTpointer;
>
>   PetscMalloc1(2,&AfullNESTpointer);
>   AfullNESTpointer[0]=NULL;
>   AfullNESTpointer[1]=NULL;
>   AfullNESTpointer[MPIrank]=Adiag;
>   // Rank=0 --> AfullNESTpointer[0]=Adiag; AfullNESTpointer[1]=NULL;
>   // Rank=1 --> AfullNESTpointer[0]=NULL;  AfullNESTpointer[1]=Adiag;
>
>   IS        IS_ROW,IS_COL;
>   ISCreateStride(PETSC_COMM_SELF,1,MPIrank,0,&IS_ROW);
>   ISCreateStride(PETSC_COMM_SELF,2,0,1,&IS_COL);
>   // Rank=0 --> IS_ROW= [ 0 ] ; IS_COL= [ 0, 1 ] ;
>   // Rank=1 --> IS_ROW= [ 1 ] ; IS_COL= [ 0, 1 ] ;
>
>   MatCreate(PETSC_COMM_WORLD,&AfullNEST);
>   MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
>   // MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
>   //
> MatCreateNest(PETSC_COMM_WORLD,1,&IS_ROW,1,&IS_COL,AfullNESTpointer,&AfullNEST);
>     ierr =
> MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
> CHKERRQ(ierr);
>
>   ierr = PetscFinalize(); CHKERRQ(ierr);
>   return 0;
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/12d3e7e6/attachment.html>

From jed at jedbrown.org  Fri Apr  5 08:25:45 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 05 Apr 2019 07:25:45 -0600
Subject: [petsc-users] PetscSFReduceBegin can not handle MPI_CHAR?
In-Reply-To: <C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
References: <CAN5Wd-Lcg0A2JFEAWsf+7MCT8YxaG0=ZpV09cX1zGdihDdMznA@mail.gmail.com>
	<CAEcYPwDdkt7a2YQnCiM3Bwf1NkVKn4cUH40DF9e4MLG8pxBd4g@mail.gmail.com>
	<CA+MQGp8C7BVbp63n1JJfZsTm02fwDSPmNEXtbDGW2uyTuUYCwQ@mail.gmail.com>
	<C6B95261-17D4-44FC-87F6-B6FEF3B4F857@gmail.com>
Message-ID: <87ef6ga8iu.fsf@jedbrown.org>

Junchao's PR has been merged to 'master'.

https://bitbucket.org/petsc/petsc/pull-requests/1511/add-signed-char-unsigned-char-and-char

Fande Kong via petsc-users <petsc-users at mcs.anl.gov> writes:

> Thanks for the reply.  It is not necessary for me to use MPI_SUM.  I think the better choice is MPIU_REPLACE. Doesn?t MPIU_REPLACE work for any mpi_datatype?
>
> Fande 
>
>
>> On Apr 3, 2019, at 9:15 PM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
>> 
>> 
>>> On Wed, Apr 3, 2019 at 3:41 AM Lisandro Dalcin via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> IIRC, MPI_CHAR is for ASCII text data. Also, remember that in C the signedness of plain `char` is implementation (or platform?) dependent.
>>>  I'm not sure MPI_Reduce() is supposed to / should  handle MPI_CHAR, you should use MPI_{SIGNED|UNSIGNED}_CHAR for that. Note however that MPI_SIGNED_CHAR is from MPI 2.0.
>> 
>> MPI standard chapter 5.9.3, says "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations"
>> So Fande's code and Jed's branch have problems. To fix that, we have to add support for signed char, unsigned char, and char in PetscSF.  The first two types support add, mult, logical and bitwise operations. The last is a dumb type, only supports pack/unpack. With this fix, PetscSF/MPI would raise error on Fande's code. I can come up with a fix tomorrow.
>>  
>>> 
>>>> On Wed, 3 Apr 2019 at 07:01, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> Hi All,
>>>> 
>>>> There were some error messages when using PetscSFReduceBegin with MPI_CHAR. 
>>>> 
>>>> ierr = PetscSFReduceBegin(ptap->sf,MPI_CHAR,rmtspace,space,MPI_SUM);CHKERRQ(ierr);
>>>> 
>>>> 
>>>> My question would be: Does PetscSFReduceBegin suppose work with MPI_CHAR? If not, should we document somewhere?
>>>> 
>>>> Thanks
>>>> 
>>>> Fande,
>>>> 
>>>> 
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>> [0]PETSC ERROR: No support for type size not divisible by 4
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [1]PETSC ERROR: No support for this operation for this object type
>>>> [1]PETSC ERROR: No support for type size not divisible by 4
>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.10.4-1989-gd816d1587e  GIT Date: 2019-04-02 17:37:18 -0600
>>>> [1]PETSC ERROR: ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
>>>> [1]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
>>>> [1]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> [1]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> [1]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> ./ex90 on a arch-linux2-c-dbg-feature-ptap-all-at-once named fn605731.local by kongf Tue Apr  2 21:48:41 2019
>>>> [0]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=yes --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 PETSC_ARCH=arch-linux2-c-dbg-feature-ptap-all-at-once --download-ptscotch --download-party --download-chaco --with-cxx-dialect=C++11
>>>> [0]PETSC ERROR: #1 PetscSFBasicPackTypeSetup() line 678 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> [0]PETSC ERROR: #2 PetscSFBasicGetPack() line 804 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> [0]PETSC ERROR: #3 PetscSFReduceBegin_Basic() line 1024 in /Users/kongf/projects/petsc/src/vec/is/sf/impls/basic/sfbasic.c
>>>> [0]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
>>>> [0]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>>> [0]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>>> [0]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
>>>> [0]PETSC ERROR: PETSc Option Table entries:
>>>> [0]PETSC ERROR: -matptap_via allatonce
>>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>>>> [1]PETSC ERROR: #4 PetscSFReduceBegin() line 1208 in /Users/kongf/projects/petsc/src/vec/is/sf/interface/sf.c
>>>> [1]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIAIJ_allatonce() line 850 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>>> [1]PETSC ERROR: #6 MatPtAP_MPIAIJ_MPIAIJ() line 202 in /Users/kongf/projects/petsc/src/mat/impls/aij/mpi/mpiptap.c
>>>> [1]PETSC ERROR: #7 MatPtAP() line 9429 in /Users/kongf/projects/petsc/src/mat/interface/matrix.c
>>>> [1]PETSC ERROR: #8 main() line 58 in /Users/kongf/projects/petsc/src/mat/examples/tests/ex90.c
>>>> [1]PETSC ERROR: PETSc Option Table entries:
>>>> [1]PETSC ERROR: -matptap_via allatonce
>>>> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>>>> --------------------------------------------------------------------------
>>>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>>>> with errorcode 56.
>>>> 
>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>> You may or may not see output from other processes, depending on
>>>> exactly when Open MPI kills them.
>>>> --------------------------------------------------------------------------
>>>> [fn605731.local:78133] 1 more process has sent help message help-mpi-api.txt / mpi-abort
>>>> [fn605731.local:78133] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
>>> 
>>> 
>>> -- 
>>> Lisandro Dalcin
>>> ============
>>> Research Scientist
>>> Extreme Computing Research Center (ECRC)
>>> King Abdullah University of Science and Technology (KAUST)
>>> http://ecrc.kaust.edu.sa/

From yyang85 at stanford.edu  Fri Apr  5 09:27:10 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 5 Apr 2019 14:27:10 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <CAMYG4G=Ax4yrfVA6cWoQ4M2KkqAOmX+thf5T22FaxJtODSM88g@mail.gmail.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
	<DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<CAMYG4G=Ax4yrfVA6cWoQ4M2KkqAOmX+thf5T22FaxJtODSM88g@mail.gmail.com>
Message-ID: <DM6PR02MB5242E5EE85E9CC9570978B5FD2510@DM6PR02MB5242.namprd02.prod.outlook.com>

Hmm ok. Then should I use this function or not when I'm reading the input? It's probably still going to give me the same error and unable to proceed?

I'd like to know if I should use something else to work around this problem.

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Friday, April 5, 2019 5:16:12 AM
To: Yuyun Yang
Cc: Smith, Barry F.; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Thu, Apr 4, 2019 at 10:56 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
So do I call MPI_Bcast right after I call PetscViewerASCIIRead? Is that going to prevent the other processors from trying to read the same file but were unable to?

No, all this does is replicate data from process 0 on the other processes.

   Matt

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Thursday, April 4, 2019 7:30:20 PM
To: Yuyun Yang
Cc: Smith, Barry F.; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
We are probably not going to use hundreds of processors, but i think it would be good to just have processor 0 read the input and broadcast that to all the other processors. Would that be a possible fix? And what would you suggest to work around this problem for now?

Explicitly call MPI_Bcast().

   Matt

Thanks!
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>
Sent: Thursday, April 4, 2019 3:07:37 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors


   Currently PetscViewerFileSetName_ASCII() only opens the file on process 0 (for read or write) thus when you call PetscViewerASCIIRead() from any process but the first it will be reading from an fd that has not been set and you could get unpredictable results.

   The implementation and documentation for PetscViewerASCIIRead() is buggy.

   There are two possible fixes we could make

1) have PetscViewerFileSetName_ASCII()  open the file for reading on all processes or
2) have PetscViewerASCIIRead() generate an error if the process is not rank == 0

   Barry

Note that using PetscViewerASCIIRead() from a handful of processes is probably fine but having hundreds or thousands of processes open the same ASCII file and reading from it will likely not be scalable.




> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hello team,
>
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>
> Thanks,
> Yuyun



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/2b66ff51/attachment-0001.html>

From knepley at gmail.com  Fri Apr  5 09:30:35 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Apr 2019 10:30:35 -0400
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <DM6PR02MB5242E5EE85E9CC9570978B5FD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
	<DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=Ax4yrfVA6cWoQ4M2KkqAOmX+thf5T22FaxJtODSM88g@mail.gmail.com>
	<DM6PR02MB5242E5EE85E9CC9570978B5FD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4Gkum05p8eo5vR3U2X5p89DL1b9hEWdg=VKgmXEpUEpe6A@mail.gmail.com>

On Fri, Apr 5, 2019 at 10:27 AM Yuyun Yang <yyang85 at stanford.edu> wrote:

> Hmm ok. Then should I use this function or not when I'm reading the input?
> It's probably still going to give me the same error and unable to proceed?
>
> I'd like to know if I should use something else to work around this
> problem.
>

No, what you do is

  if (!rank) {
    PetscViewerASCIIRead();
    MPI_Bcast();
  } else {
    MPI_Bcast();
  }

  Thanks,

     Matt


> Thanks,
> Yuyun
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, April 5, 2019 5:16:12 AM
> *To:* Yuyun Yang
> *Cc:* Smith, Barry F.; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>
> On Thu, Apr 4, 2019 at 10:56 PM Yuyun Yang <yyang85 at stanford.edu> wrote:
>
>> So do I call MPI_Bcast right after I call PetscViewerASCIIRead? Is that
>> going to prevent the other processors from trying to read the same file but
>> were unable to?
>>
>
> No, all this does is replicate data from process 0 on the other processes.
>
>    Matt
>
>
>> Thanks,
>> Yuyun
>>
>> Get Outlook for iOS <https://aka.ms/o0ukef>
>> ------------------------------
>> *From:* Matthew Knepley <knepley at gmail.com>
>> *Sent:* Thursday, April 4, 2019 7:30:20 PM
>> *To:* Yuyun Yang
>> *Cc:* Smith, Barry F.; petsc-users at mcs.anl.gov
>> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>>
>> On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> We are probably not going to use hundreds of processors, but i think it
>>> would be good to just have processor 0 read the input and broadcast that to
>>> all the other processors. Would that be a possible fix? And what would you
>>> suggest to work around this problem for now?
>>>
>>
>> Explicitly call MPI_Bcast().
>>
>>    Matt
>>
>>
>>> Thanks!
>>> Yuyun
>>>
>>> Get Outlook for iOS <https://aka.ms/o0ukef>
>>> ------------------------------
>>> *From:* Smith, Barry F. <bsmith at mcs.anl.gov>
>>> *Sent:* Thursday, April 4, 2019 3:07:37 PM
>>> *To:* Yuyun Yang
>>> *Cc:* petsc-users at mcs.anl.gov
>>> *Subject:* Re: [petsc-users] ASCIIRead error for multiple processors
>>>
>>>
>>>    Currently PetscViewerFileSetName_ASCII() only opens the file on
>>> process 0 (for read or write) thus when you call PetscViewerASCIIRead()
>>> from any process but the first it will be reading from an fd that has not
>>> been set and you could get unpredictable results.
>>>
>>>    The implementation and documentation for PetscViewerASCIIRead() is
>>> buggy.
>>>
>>>    There are two possible fixes we could make
>>>
>>> 1) have PetscViewerFileSetName_ASCII()  open the file for reading on all
>>> processes or
>>> 2) have PetscViewerASCIIRead() generate an error if the process is not
>>> rank == 0
>>>
>>>    Barry
>>>
>>> Note that using PetscViewerASCIIRead() from a handful of processes is
>>> probably fine but having hundreds or thousands of processes open the same
>>> ASCII file and reading from it will likely not be scalable.
>>>
>>>
>>>
>>>
>>> > On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >
>>> > Hello team,
>>> >
>>> > I?m trying to use PetscViewerASCIIRead() to read in a single
>>> interger/scalar value from an input file. It works for one processor.
>>> However, when running on multiple processors, I?m getting the below error:
>>> >
>>> > [1]PETSC ERROR: Invalid argument
>>> > [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
>>> > [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in
>>> /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>>> >
>>> > Is there something wrong with how I?m implementing this, or ASCIIRead
>>> does not work with multiple processors?
>>> >
>>> > Thanks,
>>> > Yuyun
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/972a2129/attachment.html>

From yyang85 at stanford.edu  Fri Apr  5 10:37:01 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 5 Apr 2019 15:37:01 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <CAMYG4Gkum05p8eo5vR3U2X5p89DL1b9hEWdg=VKgmXEpUEpe6A@mail.gmail.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
	<3D59F6EF-AE91-46D4-B5E6-2D4BFC454957@anl.gov>
	<DM6PR02MB5242C9D895FA267B3C0B4E44D2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=qXqoUybx_ooTr4YpRhWxvYtSHAueSB7TMh6Jbm8LgZA@mail.gmail.com>
	<DM6PR02MB52421956ACBF4BDF8D06961DD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=Ax4yrfVA6cWoQ4M2KkqAOmX+thf5T22FaxJtODSM88g@mail.gmail.com>
	<DM6PR02MB5242E5EE85E9CC9570978B5FD2510@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4Gkum05p8eo5vR3U2X5p89DL1b9hEWdg=VKgmXEpUEpe6A@mail.gmail.com>
Message-ID: <DM6PR02MB524270DBFA87E73E872AA8BFD2510@DM6PR02MB5242.namprd02.prod.outlook.com>

Ok great, thanks for the explanation!

Best,
Yuyun

From: Matthew Knepley <knepley at gmail.com>
Sent: Friday, April 5, 2019 7:31 AM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: Smith, Barry F. <bsmith at mcs.anl.gov>; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Fri, Apr 5, 2019 at 10:27 AM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Hmm ok. Then should I use this function or not when I'm reading the input? It's probably still going to give me the same error and unable to proceed?

I'd like to know if I should use something else to work around this problem.

No, what you do is

  if (!rank) {
    PetscViewerASCIIRead();
    MPI_Bcast();
  } else {
    MPI_Bcast();
  }

  Thanks,

     Matt

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, April 5, 2019 5:16:12 AM
To: Yuyun Yang
Cc: Smith, Barry F.; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Thu, Apr 4, 2019 at 10:56 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
So do I call MPI_Bcast right after I call PetscViewerASCIIRead? Is that going to prevent the other processors from trying to read the same file but were unable to?

No, all this does is replicate data from process 0 on the other processes.

   Matt

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Thursday, April 4, 2019 7:30:20 PM
To: Yuyun Yang
Cc: Smith, Barry F.; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors

On Thu, Apr 4, 2019 at 9:19 PM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
We are probably not going to use hundreds of processors, but i think it would be good to just have processor 0 read the input and broadcast that to all the other processors. Would that be a possible fix? And what would you suggest to work around this problem for now?

Explicitly call MPI_Bcast().

   Matt

Thanks!
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>
Sent: Thursday, April 4, 2019 3:07:37 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] ASCIIRead error for multiple processors


   Currently PetscViewerFileSetName_ASCII() only opens the file on process 0 (for read or write) thus when you call PetscViewerASCIIRead() from any process but the first it will be reading from an fd that has not been set and you could get unpredictable results.

   The implementation and documentation for PetscViewerASCIIRead() is buggy.

   There are two possible fixes we could make

1) have PetscViewerFileSetName_ASCII()  open the file for reading on all processes or
2) have PetscViewerASCIIRead() generate an error if the process is not rank == 0

   Barry

Note that using PetscViewerASCIIRead() from a handful of processes is probably fine but having hundreds or thousands of processes open the same ASCII file and reading from it will likely not be scalable.




> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hello team,
>
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>
> Thanks,
> Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/62e0df3d/attachment-0001.html>

From maahi.buet at gmail.com  Fri Apr  5 15:32:24 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Fri, 5 Apr 2019 16:32:24 -0400
Subject: [petsc-users] Solution Diverging
Message-ID: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>

Dear All,

I am solving a linear system arising from 9-point discretization of Poisson
equation to generate grid. KSP solvers are giving nice converged result
when I am dealing with a large matrix. But when I am trying to implement
KSP solvers when the matrix is smaller and the values are smaller too, the
solution is diverging.
Any suggestions how to get around this problem?


Regards,
Maahi Talukder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/a28d7db9/attachment.html>

From danyang.su at gmail.com  Fri Apr  5 15:42:53 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Fri, 5 Apr 2019 13:42:53 -0700
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
Message-ID: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>

Hi All,

I got a strange error in calling DMPlexDistribute after updating PETSc 
to V3.11.0. There sounds no change in the interface of DMPlexDistribute 
as documented in

https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute

https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute

The code section is shown below.

 ????? !c distribute mesh over processes
 ????? call 
DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)

When I use PETSc V3.10 and earlier versions, it works fine. After 
updating to latest PETSc V3.11.0, I got the following error during compiling

 ????? call 
DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
 ???????????????????????????????????????????????????????????????????? 1
Error: Non-variable expression in variable definition context (actual 
argument to INTENT = OUT/INOUT) at (1)
/home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31: 
recipe for target '../../solver/solver_ddmethod.o' failed

The fortran example 
/home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90 
which also uses DMPlexDistribute can be compiled without problem. Is 
there any updates in the compiler flags I need to change?

Thanks,

Danyang



From balay at mcs.anl.gov  Fri Apr  5 15:55:55 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 5 Apr 2019 20:55:55 +0000
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
Message-ID: <alpine.LFD.2.21.1904051549490.2802@sb>

A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.

http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048

+          PetscSF, intent(out) :: sf

So this should be inout?

[All auto-generated stubs don't quantify intent()]

Satish

On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:

> Hi All,
> 
> I got a strange error in calling DMPlexDistribute after updating PETSc to
> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
> documented in
> 
> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> 
> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> 
> The code section is shown below.
> 
> ????? !c distribute mesh over processes
> ????? call 
> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> 
> When I use PETSc V3.10 and earlier versions, it works fine. After updating to
> latest PETSc V3.11.0, I got the following error during compiling
> 
> ????? call
> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> ???????????????????????????????????????????????????????????????????? 1
> Error: Non-variable expression in variable definition context (actual argument
> to INTENT = OUT/INOUT) at (1)
> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
> recipe for target '../../solver/solver_ddmethod.o' failed
> 
> The fortran example
> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
> which also uses DMPlexDistribute can be compiled without problem. Is there any
> updates in the compiler flags I need to change?
> 
> Thanks,
> 
> Danyang
> 
> 
> 

From balay at mcs.anl.gov  Fri Apr  5 16:00:20 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 5 Apr 2019 21:00:20 +0000
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <alpine.LFD.2.21.1904051549490.2802@sb>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
Message-ID: <alpine.LFD.2.21.1904051557500.2802@sb>

Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base here..

Satish


On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:

> A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.
> 
> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
> 
> +          PetscSF, intent(out) :: sf
> 
> So this should be inout?
> 
> [All auto-generated stubs don't quantify intent()]
> 
> Satish
> 
> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
> 
> > Hi All,
> > 
> > I got a strange error in calling DMPlexDistribute after updating PETSc to
> > V3.11.0. There sounds no change in the interface of DMPlexDistribute as
> > documented in
> > 
> > https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> > 
> > https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> > 
> > The code section is shown below.
> > 
> > ????? !c distribute mesh over processes
> > ????? call 
> > DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> > 
> > When I use PETSc V3.10 and earlier versions, it works fine. After updating to
> > latest PETSc V3.11.0, I got the following error during compiling
> > 
> > ????? call
> > DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> > ???????????????????????????????????????????????????????????????????? 1
> > Error: Non-variable expression in variable definition context (actual argument
> > to INTENT = OUT/INOUT) at (1)
> > /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
> > recipe for target '../../solver/solver_ddmethod.o' failed
> > 
> > The fortran example
> > /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
> > which also uses DMPlexDistribute can be compiled without problem. Is there any
> > updates in the compiler flags I need to change?
> > 
> > Thanks,
> > 
> > Danyang
> > 
> > 
> > 
> 

From danyang.su at gmail.com  Fri Apr  5 16:05:44 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Fri, 5 Apr 2019 14:05:44 -0700
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <alpine.LFD.2.21.1904051557500.2802@sb>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
	<alpine.LFD.2.21.1904051557500.2802@sb>
Message-ID: <80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>

Hi Satish,

Because of the intent(out) declaration, I use a temporary solution that 
passing a PetscSF type variable to DMPlexDistribute instead of passing 
PETSC_NULL_SF. But I am still confused how the example works without 
problem.

Thanks,

Danyang

On 2019-04-05 2:00 p.m., Balay, Satish wrote:
> Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base here..
>
> Satish
>
>
> On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:
>
>> A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.
>>
>> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
>>
>> +          PetscSF, intent(out) :: sf
>>
>> So this should be inout?
>>
>> [All auto-generated stubs don't quantify intent()]
>>
>> Satish
>>
>> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
>>
>>> Hi All,
>>>
>>> I got a strange error in calling DMPlexDistribute after updating PETSc to
>>> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
>>> documented in
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>
>>> The code section is shown below.
>>>
>>>  ????? !c distribute mesh over processes
>>>  ????? call
>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>
>>> When I use PETSc V3.10 and earlier versions, it works fine. After updating to
>>> latest PETSc V3.11.0, I got the following error during compiling
>>>
>>>  ????? call
>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>  ???????????????????????????????????????????????????????????????????? 1
>>> Error: Non-variable expression in variable definition context (actual argument
>>> to INTENT = OUT/INOUT) at (1)
>>> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>>> recipe for target '../../solver/solver_ddmethod.o' failed
>>>
>>> The fortran example
>>> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>>> which also uses DMPlexDistribute can be compiled without problem. Is there any
>>> updates in the compiler flags I need to change?
>>>
>>> Thanks,
>>>
>>> Danyang
>>>
>>>
>>>

From balay at mcs.anl.gov  Fri Apr  5 16:13:00 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 5 Apr 2019 21:13:00 +0000
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
	<alpine.LFD.2.21.1904051557500.2802@sb>
	<80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
Message-ID: <alpine.LFD.2.21.1904051610540.2802@sb>

A complete simple code showing this error would be useful.

wrt 'Error: Non-variable expression in variable definition context' google gives:

https://www.queryxchange.com/q/27_45961166/non-variable-expression-in-variable-definition-context-compilation-error/

But I don't see an expression in the code snippet here..

There must be some difference between your usage of this function vs the petsc example..

Satish

On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:

> Hi Satish,
> 
> Because of the intent(out) declaration, I use a temporary solution that
> passing a PetscSF type variable to DMPlexDistribute instead of passing
> PETSC_NULL_SF. But I am still confused how the example works without problem.
> 
> Thanks,
> 
> Danyang
> 
> On 2019-04-05 2:00 p.m., Balay, Satish wrote:
> > Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base
> > here..
> >
> > Satish
> >
> >
> > On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:
> >
> >> A fortran interface definition was added in petsc-3.11 so the compile now
> >> checks if its used correctly.
> >>
> >> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
> >>
> >> +          PetscSF, intent(out) :: sf
> >>
> >> So this should be inout?
> >>
> >> [All auto-generated stubs don't quantify intent()]
> >>
> >> Satish
> >>
> >> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
> >>
> >>> Hi All,
> >>>
> >>> I got a strange error in calling DMPlexDistribute after updating PETSc to
> >>> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
> >>> documented in
> >>>
> >>> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> >>>
> >>> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
> >>>
> >>> The code section is shown below.
> >>>
> >>>  ????? !c distribute mesh over processes
> >>>  ????? call
> >>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> >>>
> >>> When I use PETSc V3.10 and earlier versions, it works fine. After updating
> >>> to
> >>> latest PETSc V3.11.0, I got the following error during compiling
> >>>
> >>> ????? call
> >>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
> >>> ???????????????????????????????????????????????????????????????????? 1
> >>> Error: Non-variable expression in variable definition context (actual
> >>> argument
> >>> to INTENT = OUT/INOUT) at (1)
> >>> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
> >>> recipe for target '../../solver/solver_ddmethod.o' failed
> >>>
> >>> The fortran example
> >>> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
> >>> which also uses DMPlexDistribute can be compiled without problem. Is there
> >>> any
> >>> updates in the compiler flags I need to change?
> >>>
> >>> Thanks,
> >>>
> >>> Danyang
> >>>
> >>>
> >>>
> 

From mfadams at lbl.gov  Fri Apr  5 16:21:10 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 5 Apr 2019 17:21:10 -0400
Subject: [petsc-users] Solution Diverging
In-Reply-To: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>
References: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>
Message-ID: <CADOhEh7mbBbRj=uR9KeTJW3-p-jEUHfNXb_N3-Rj-87z_xeCUQ@mail.gmail.com>

So you have a 2D Poisson solver and it is working on large problems but the
linear solver is diverging on small problems.

I would verify the the solution is good on the large problem.

You might have a problem with boundary conditions. What BCs do you think
you are using?

On Fri, Apr 5, 2019 at 4:33 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear All,
>
> I am solving a linear system arising from 9-point discretization of
> Poisson equation to generate grid. KSP solvers are giving nice converged
> result when I am dealing with a large matrix. But when I am trying to
> implement KSP solvers when the matrix is smaller and the values are smaller
> too, the solution is diverging.
> Any suggestions how to get around this problem?
>
>
> Regards,
> Maahi Talukder
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/54a4a371/attachment.html>

From mfadams at lbl.gov  Fri Apr  5 17:23:04 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 5 Apr 2019 18:23:04 -0400
Subject: [petsc-users] Solution Diverging
In-Reply-To: <CAJOaupErEd2yUodgM3p1UpVMe_P4e-UjLQFL9hC74uUrp_hzfA@mail.gmail.com>
References: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>
	<CADOhEh7mbBbRj=uR9KeTJW3-p-jEUHfNXb_N3-Rj-87z_xeCUQ@mail.gmail.com>
	<CAJOaupErEd2yUodgM3p1UpVMe_P4e-UjLQFL9hC74uUrp_hzfA@mail.gmail.com>
Message-ID: <CADOhEh68b3gaqfCng5tHW6eF1R0GYOu5DOg-wvSD8Ye8sbfx7A@mail.gmail.com>

On Fri, Apr 5, 2019 at 5:27 PM Maahi Talukder <maahi.buet at gmail.com> wrote:

> Hi,
> Thank you for your reply.
>
> Well, I have verified the solution on large problems, and I have converged
> the solution to machine accuracy using different solvers. So I guess the
> setup okay in that respect.
>

I am suspicious that it "worked" on large problems but failed on small. But
you never know.


>
> The BC I am using is of Dirichlet type.
>
>
>
> On Fri, Apr 5, 2019 at 5:21 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> So you have a 2D Poisson solver and it is working on large problems but
>> the linear solver is diverging on small problems.
>>
>> I would verify the the solution is good on the large problem.
>>
>> You might have a problem with boundary conditions. What BCs do you think
>> you are using?
>>
>> On Fri, Apr 5, 2019 at 4:33 PM Maahi Talukder via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Dear All,
>>>
>>> I am solving a linear system arising from 9-point discretization of
>>> Poisson equation to generate grid. KSP solvers are giving nice converged
>>> result when I am dealing with a large matrix. But when I am trying to
>>> implement KSP solvers when the matrix is smaller and the values are smaller
>>> too, the solution is diverging.
>>> Any suggestions how to get around this problem?
>>>
>>>
>>> Regards,
>>> Maahi Talukder
>>>
>>>
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/41421bf1/attachment.html>

From knepley at gmail.com  Fri Apr  5 20:00:20 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Apr 2019 21:00:20 -0400
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
Message-ID: <CAMYG4G=BCx1+Yu9J4z-v++JOZM0AQdD0f1U25s6p2FUctgquwg@mail.gmail.com>

On Fri, Apr 5, 2019 at 4:44 PM Danyang Su via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi All,
>
> I got a strange error in calling DMPlexDistribute after updating PETSc
> to V3.11.0. There sounds no change in the interface of DMPlexDistribute
> as documented in
>
>
> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>
>
> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>
> The code section is shown below.
>
>        !c distribute mesh over processes
>        call
> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>
> When I use PETSc V3.10 and earlier versions, it works fine. After
> updating to latest PETSc V3.11.0, I got the following error during
> compiling
>
>        call
> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>                                                                       1
> Error: Non-variable expression in variable definition context (actual
> argument to INTENT = OUT/INOUT) at (1)
> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>
> recipe for target '../../solver/solver_ddmethod.o' failed
>
> The fortran example
> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>
> which also uses DMPlexDistribute can be compiled without problem. Is
> there any updates in the compiler flags I need to change?
>

What includes do you have? It looks like it does not understand
PETSC_NULL_SF.

  Thanks,

    Matt


> Thanks,
>
> Danyang
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/46c40db3/attachment.html>

From knepley at gmail.com  Fri Apr  5 20:05:07 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Apr 2019 21:05:07 -0400
Subject: [petsc-users] Solution Diverging
In-Reply-To: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>
References: <CAJOaupGqjGRn0kpze0_QMcFkD-1wiWZExdUHxSvOOXZ5KH0m=w@mail.gmail.com>
Message-ID: <CAMYG4G=-n51ihZfKPTbWBN690vMEoYVWutn4n9VH2hEg+bLPCg@mail.gmail.com>

On Fri, Apr 5, 2019 at 4:33 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear All,
>
> I am solving a linear system arising from 9-point discretization of
> Poisson equation to generate grid. KSP solvers are giving nice converged
> result when I am dealing with a large matrix. But when I am trying to
> implement KSP solvers when the matrix is smaller and the values are smaller
> too, the solution is diverging.
> Any suggestions how to get around this problem?
>

For any solver question, send us the output from

  -ksp_view -ksp_monitor_true_residual -ksp_converged_reason

Divergence is surprising since the default Krylov solver (GMRES) cannot
diverge.

  Thanks,

    Matt


> Regards,
> Maahi Talukder
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/5432a313/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Fri Apr  5 20:32:49 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 5 Apr 2019 20:32:49 -0500
Subject: [petsc-users] Estimate memory needs for large grids
Message-ID: <CAOGsD9iPfJFFf7t2YpRijnUX_RLXXCZGLO5NeVw2mRT_kAR8Gw@mail.gmail.com>

Hi,

I've solving a simple linear equation [ u_t = A*u_xx + A*u_yy + F_t*u ] on
a grid size of 55296x55296. I'm reading a vector of that size from an hdf5
file and have the jacobian matrix as a modified 5-point stencil which is
preallocated with the following
```
  ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,M,M);CHKERRQ(ierr);
  ierr = MatSetType(A,MATMPIAIJ);CHKERRQ(ierr);
  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
  ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);
  ierr = MatSeqAIJSetPreallocation(A,5,NULL);CHKERRQ(ierr);
```
Total number of elements is ~3e9 and the matrix size is ~9e9 (but only 5
diagonals are non zeros). I'm reading F_t which has ~3e9 elements. I'm
using double complex numbers and I've compiled with int64 indices.

Thus, for the vector I need, 55296x55296x2x8 bytes ~ 50Gb and for the F
vector, another 50 Gb. For the matrix I need ~250 Gb and some overhead for
the solver.

How do I estimate this overhead (and estimate how many nodes I would need
to run this given the maximum memory per node (as specified by slurm's
--mem option)) ?

Thanks in advance for the help!

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190405/ab107adb/attachment-0001.html>

From jed at jedbrown.org  Fri Apr  5 22:52:55 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 05 Apr 2019 21:52:55 -0600
Subject: [petsc-users] Estimate memory needs for large grids
In-Reply-To: <CAOGsD9iPfJFFf7t2YpRijnUX_RLXXCZGLO5NeVw2mRT_kAR8Gw@mail.gmail.com>
References: <CAOGsD9iPfJFFf7t2YpRijnUX_RLXXCZGLO5NeVw2mRT_kAR8Gw@mail.gmail.com>
Message-ID: <87tvfb7pt4.fsf@jedbrown.org>

Memory use will depend on the preconditioner.  This will converge very
slowly (i.e., never) without multigrid unless time steps are small.
Depending on how rough the coefficients are, you may be able to use
geometric multigrid, which has pretty low setup costs and memory
requirements.

To estimate memory with an arbitrary preconditioner, I would run a
smaller problem using the desired preconditioner and check its memory
use using -log_view.  From that you can estimate total memory
requirements for the target job.

Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi,
>
> I've solving a simple linear equation [ u_t = A*u_xx + A*u_yy + F_t*u ] on
> a grid size of 55296x55296. I'm reading a vector of that size from an hdf5
> file and have the jacobian matrix as a modified 5-point stencil which is
> preallocated with the following
> ```
>   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>   ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,M,M);CHKERRQ(ierr);
>   ierr = MatSetType(A,MATMPIAIJ);CHKERRQ(ierr);
>   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>   ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);
>   ierr = MatSeqAIJSetPreallocation(A,5,NULL);CHKERRQ(ierr);
> ```
> Total number of elements is ~3e9 and the matrix size is ~9e9 (but only 5
> diagonals are non zeros). I'm reading F_t which has ~3e9 elements. I'm
> using double complex numbers and I've compiled with int64 indices.
>
> Thus, for the vector I need, 55296x55296x2x8 bytes ~ 50Gb and for the F
> vector, another 50 Gb. For the matrix I need ~250 Gb and some overhead for
> the solver.
>
> How do I estimate this overhead (and estimate how many nodes I would need
> to run this given the maximum memory per node (as specified by slurm's
> --mem option)) ?
>
> Thanks in advance for the help!
>
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University

From bsmith at mcs.anl.gov  Sat Apr  6 01:29:39 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 6 Apr 2019 06:29:39 +0000
Subject: [petsc-users] error: Petsc has generated inconsistent data,
 MPI_Allreduce() called in different locations (code lines) on
 different processors
In-Reply-To: <CAMYG4Gk8_XSH9YSXR4Fmv8EPcRMYhiRHYjyo3F0Tbav6bdF9Gg@mail.gmail.com>
References: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>
	<CAMYG4Gk8_XSH9YSXR4Fmv8EPcRMYhiRHYjyo3F0Tbav6bdF9Gg@mail.gmail.com>
Message-ID: <392602E7-E4F5-4B43-9079-41DEB1BB857B@anl.gov>


  Eda,

   Can you send us your code (and any needed data files)? We certainly expect PETSc to perform correctly if the size of the matrix cannot be divide by the number of processors. It is possible the problem is due to bugs either in MatStashScatterBegin_BTS() or in your code.

   Thanks

    Barry


> On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello,
> 
> I am trying to calculate unweighted Laplacian of a matrix by using 2 cores. If the size of matrix is in even number then my program works. However, when I try to use a matrix having odd number for size, I guess since size of the matrix cannot be divided into processors correctly, I get the following error:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Petsc has generated inconsistent data
> [1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5 09:50:54 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5 09:50:54 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #1 MatStashScatterBegin_BTS() line 843 in /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #2 MatStashScatterBegin_BTS() line 843 in /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [0]PETSC ERROR: #3 main() line 164 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> #5 MatAssemblyBegin() line 5251 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> #6 main() line 169 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -unweighted
> [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
> 
> 
>  where line 164 in my main program is MatSetOption and line 169 is MatAssemblyBegin. I am new to MPI usage so I did not understand why MPI_Allreduce() causes a problem and how can I fix the problem.
> 
> You have to call collective methods on all processes in the same order. This is not happening in your code. Beyond that,
> there is no way for us to tell how this happened.
> 
>   Thanks,
> 
>      Matt
>  
> Thank you,
> 
> Eda
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From bsmith at mcs.anl.gov  Sat Apr  6 02:13:51 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 6 Apr 2019 07:13:51 +0000
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
	<alpine.LFD.2.21.1904051557500.2802@sb>
	<80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
Message-ID: <154F39A1-C04E-4FAD-A0C8-3A0A9C3B1DCF@anl.gov>



> On Apr 5, 2019, at 4:05 PM, Danyang Su via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi Satish,
> 
> Because of the intent(out) declaration, I use a temporary solution that passing a PetscSF type variable to DMPlexDistribute instead of passing PETSC_NULL_SF. But I am still confused how the example works without problem.

   Yes this is strange. I cannot explain the inconsistent behavior between the two codes. Any chance you can send us your code (petsc-maint at mcs.anl.gov) so we can reproduce the problem?

   The intent(out) is obviously wrong in the interface definition since the function need not return the value. Could you please try removing the intent from the interface definition in http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048, run make again on PETSc and see if that allows you to use PETSC_NULL_SF in your code? 

   We would like to fix PETSc to handle this case

   Thanks

     Barry

> 
> Thanks,
> 
> Danyang
> 
> On 2019-04-05 2:00 p.m., Balay, Satish wrote:
>> Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base here..
>> 
>> Satish
>> 
>> 
>> On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:
>> 
>>> A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.
>>> 
>>> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
>>> 
>>> +          PetscSF, intent(out) :: sf
>>> 
>>> So this should be inout?
>>> 
>>> [All auto-generated stubs don't quantify intent()]
>>> 
>>> Satish
>>> 
>>> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
>>> 
>>>> Hi All,
>>>> 
>>>> I got a strange error in calling DMPlexDistribute after updating PETSc to
>>>> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
>>>> documented in
>>>> 
>>>> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>> 
>>>> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>> 
>>>> The code section is shown below.
>>>> 
>>>>       !c distribute mesh over processes
>>>>       call
>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>> 
>>>> When I use PETSc V3.10 and earlier versions, it works fine. After updating to
>>>> latest PETSc V3.11.0, I got the following error during compiling
>>>> 
>>>>       call
>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>>                                                                      1
>>>> Error: Non-variable expression in variable definition context (actual argument
>>>> to INTENT = OUT/INOUT) at (1)
>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>>>> recipe for target '../../solver/solver_ddmethod.o' failed
>>>> 
>>>> The fortran example
>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>>>> which also uses DMPlexDistribute can be compiled without problem. Is there any
>>>> updates in the compiler flags I need to change?
>>>> 
>>>> Thanks,
>>>> 
>>>> Danyang
>>>> 
>>>> 
>>>> 


From danyang.su at gmail.com  Sat Apr  6 12:33:34 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Sat, 6 Apr 2019 10:33:34 -0700
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <CAMYG4G=BCx1+Yu9J4z-v++JOZM0AQdD0f1U25s6p2FUctgquwg@mail.gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<CAMYG4G=BCx1+Yu9J4z-v++JOZM0AQdD0f1U25s6p2FUctgquwg@mail.gmail.com>
Message-ID: <8b6927b8-c8f6-0ca8-9530-7bc0e95cdd4e@gmail.com>


On 2019-04-05 6:00 p.m., Matthew Knepley wrote:
> On Fri, Apr 5, 2019 at 4:44 PM Danyang Su via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Hi All,
>
>     I got a strange error in calling DMPlexDistribute after updating
>     PETSc
>     to V3.11.0. There sounds no change in the interface of
>     DMPlexDistribute
>     as documented in
>
>     https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>
>     https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>
>     The code section is shown below.
>
>     ?????? !c distribute mesh over processes
>     ?????? call
>     DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>
>     When I use PETSc V3.10 and earlier versions, it works fine. After
>     updating to latest PETSc V3.11.0, I got the following error during
>     compiling
>
>     ?????? call
>     DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>     ?????????????????????????????????????????????????????????????????????
>     1
>     Error: Non-variable expression in variable definition context (actual
>     argument to INTENT = OUT/INOUT) at (1)
>     /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>
>     recipe for target '../../solver/solver_ddmethod.o' failed
>
>     The fortran example
>     /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>
>     which also uses DMPlexDistribute can be compiled without problem. Is
>     there any updates in the compiler flags I need to change?
>
>
> What includes do you have? It looks like it does not understand 
> PETSC_NULL_SF.
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petsc/finclude/petscdmlabel.h>

It looks like DMPlexDistribute requires the third parameter to be in/out 
declaration while PETSC_NULL_SF does not allow to do this.

Danyang

>
> ? Thanks,
>
> ? ? Matt
>
>     Thanks,
>
>     Danyang
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190406/56d2f24b/attachment.html>

From danyang.su at gmail.com  Sat Apr  6 13:33:29 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Sat, 6 Apr 2019 11:33:29 -0700
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <154F39A1-C04E-4FAD-A0C8-3A0A9C3B1DCF@anl.gov>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
	<alpine.LFD.2.21.1904051557500.2802@sb>
	<80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
	<154F39A1-C04E-4FAD-A0C8-3A0A9C3B1DCF@anl.gov>
Message-ID: <6e213095-04fb-1288-c708-eefc801699b7@gmail.com>


On 2019-04-06 12:13 a.m., Smith, Barry F. wrote:
>
>> On Apr 5, 2019, at 4:05 PM, Danyang Su via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi Satish,
>>
>> Because of the intent(out) declaration, I use a temporary solution that passing a PetscSF type variable to DMPlexDistribute instead of passing PETSC_NULL_SF. But I am still confused how the example works without problem.
>     Yes this is strange. I cannot explain the inconsistent behavior between the two codes. Any chance you can send us your code (petsc-maint at mcs.anl.gov) so we can reproduce the problem?
>
>     The intent(out) is obviously wrong in the interface definition since the function need not return the value. Could you please try removing the intent from the interface definition in http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048, run make again on PETSc and see if that allows you to use PETSC_NULL_SF in your code?
>
>     We would like to fix PETSc to handle this case
>
>     Thanks
>
>       Barry

Hi Barry,

After changing "PetscSF, intent(out) :: sf" to "PetscSF :: sf" in 
DMPlexDistribute interface, it allows me to use PETSC_NULL_SF in my code.

Thanks,

Danyang

>> Thanks,
>>
>> Danyang
>>
>> On 2019-04-05 2:00 p.m., Balay, Satish wrote:
>>> Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base here..
>>>
>>> Satish
>>>
>>>
>>> On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:
>>>
>>>> A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.
>>>>
>>>> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
>>>>
>>>> +          PetscSF, intent(out) :: sf
>>>>
>>>> So this should be inout?
>>>>
>>>> [All auto-generated stubs don't quantify intent()]
>>>>
>>>> Satish
>>>>
>>>> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I got a strange error in calling DMPlexDistribute after updating PETSc to
>>>>> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
>>>>> documented in
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>>>
>>>>> The code section is shown below.
>>>>>
>>>>>        !c distribute mesh over processes
>>>>>        call
>>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>>>
>>>>> When I use PETSc V3.10 and earlier versions, it works fine. After updating to
>>>>> latest PETSc V3.11.0, I got the following error during compiling
>>>>>
>>>>>        call
>>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>>>                                                                       1
>>>>> Error: Non-variable expression in variable definition context (actual argument
>>>>> to INTENT = OUT/INOUT) at (1)
>>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>>>>> recipe for target '../../solver/solver_ddmethod.o' failed
>>>>>
>>>>> The fortran example
>>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>>>>> which also uses DMPlexDistribute can be compiled without problem. Is there any
>>>>> updates in the compiler flags I need to change?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Danyang
>>>>>
>>>>>
>>>>>

From bsmith at mcs.anl.gov  Sat Apr  6 13:53:56 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 6 Apr 2019 18:53:56 +0000
Subject: [petsc-users] Strange compiling error in DMPlexDistribute after
 updating PETSc to V3.11.0
In-Reply-To: <6e213095-04fb-1288-c708-eefc801699b7@gmail.com>
References: <5fdcc6ad-f971-5bb0-4d84-6e6e8c3dcf0c@gmail.com>
	<alpine.LFD.2.21.1904051549490.2802@sb>
	<alpine.LFD.2.21.1904051557500.2802@sb>
	<80d22a9f-c722-67d7-8007-d4b6e31e4964@gmail.com>
	<154F39A1-C04E-4FAD-A0C8-3A0A9C3B1DCF@anl.gov>
	<6e213095-04fb-1288-c708-eefc801699b7@gmail.com>
Message-ID: <0FA16A2F-A4FC-4540-A3E6-50E380AD9997@mcs.anl.gov>


   Great, thanks for letting us know. This will be fixed in our next 3.11.1 release https://bitbucket.org/petsc/petsc/pull-requests/1513/dmplexdistribute-sf-argument-need-not-be/diff

   Barry


> On Apr 6, 2019, at 1:33 PM, Danyang Su <danyang.su at gmail.com> wrote:
> 
> 
> On 2019-04-06 12:13 a.m., Smith, Barry F. wrote:
>> 
>>> On Apr 5, 2019, at 4:05 PM, Danyang Su via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi Satish,
>>> 
>>> Because of the intent(out) declaration, I use a temporary solution that passing a PetscSF type variable to DMPlexDistribute instead of passing PETSC_NULL_SF. But I am still confused how the example works without problem.
>>    Yes this is strange. I cannot explain the inconsistent behavior between the two codes. Any chance you can send us your code (petsc-maint at mcs.anl.gov) so we can reproduce the problem?
>> 
>>    The intent(out) is obviously wrong in the interface definition since the function need not return the value. Could you please try removing the intent from the interface definition in http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048, run make again on PETSc and see if that allows you to use PETSC_NULL_SF in your code?
>> 
>>    We would like to fix PETSc to handle this case
>> 
>>    Thanks
>> 
>>      Barry
> 
> Hi Barry,
> 
> After changing "PetscSF, intent(out) :: sf" to "PetscSF :: sf" in DMPlexDistribute interface, it allows me to use PETSC_NULL_SF in my code.
> 
> Thanks,
> 
> Danyang
> 
>>> Thanks,
>>> 
>>> Danyang
>>> 
>>> On 2019-04-05 2:00 p.m., Balay, Satish wrote:
>>>> Ah - the message about distributed_dm - not PETSC_NULL_SF. So I'm off base here..
>>>> 
>>>> Satish
>>>> 
>>>> 
>>>> On Fri, 5 Apr 2019, Balay, Satish via petsc-users wrote:
>>>> 
>>>>> A fortran interface definition was added in petsc-3.11 so the compile now checks if its used correctly.
>>>>> 
>>>>> http://bitbucket.org/petsc/petsc/commits/fdb49207a8b58c421782c7e45b1394c0a6567048
>>>>> 
>>>>> +          PetscSF, intent(out) :: sf
>>>>> 
>>>>> So this should be inout?
>>>>> 
>>>>> [All auto-generated stubs don't quantify intent()]
>>>>> 
>>>>> Satish
>>>>> 
>>>>> On Fri, 5 Apr 2019, Danyang Su via petsc-users wrote:
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I got a strange error in calling DMPlexDistribute after updating PETSc to
>>>>>> V3.11.0. There sounds no change in the interface of DMPlexDistribute as
>>>>>> documented in
>>>>>> 
>>>>>> https://www.mcs.anl.gov/petsc/petsc-3.10/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>>>> 
>>>>>> https://www.mcs.anl.gov/petsc/petsc-3.11/docs/manualpages/DMPLEX/DMPlexDistribute.html#DMPlexDistribute
>>>>>> 
>>>>>> The code section is shown below.
>>>>>> 
>>>>>>       !c distribute mesh over processes
>>>>>>       call
>>>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>>>> 
>>>>>> When I use PETSc V3.10 and earlier versions, it works fine. After updating to
>>>>>> latest PETSc V3.11.0, I got the following error during compiling
>>>>>> 
>>>>>>       call
>>>>>> DMPlexDistribute(dm,stencil_width,PETSC_NULL_SF,distributed_dm,ierr)
>>>>>>                                                                      1
>>>>>> Error: Non-variable expression in variable definition context (actual argument
>>>>>> to INTENT = OUT/INOUT) at (1)
>>>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/linux-gnu-opt/lib/petsc/conf/petscrules:31:
>>>>>> recipe for target '../../solver/solver_ddmethod.o' failed
>>>>>> 
>>>>>> The fortran example
>>>>>> /home/dsu/Soft/PETSc/petsc-3.11.0/src/dm/label/examples/tutorials/ex1f90.F90
>>>>>> which also uses DMPlexDistribute can be compiled without problem. Is there any
>>>>>> updates in the compiler flags I need to change?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Danyang
>>>>>> 
>>>>>> 
>>>>>> 


From zulfi6000 at yahoo.com  Sat Apr  6 19:48:10 2019
From: zulfi6000 at yahoo.com (Zulfi Khan)
Date: Sun, 7 Apr 2019 00:48:10 +0000 (UTC)
Subject: [petsc-users] All-to-All Personalized Communication on a Mesh
References: <1413664627.748377.1554598090343.ref@mail.yahoo.com>
Message-ID: <1413664627.748377.1554598090343@mail.yahoo.com>

Hi,Can some body please tell me what would??be the result of applying?All-to-All Personalized Communication on a Mesh shown on slide 54 on the following link:
https://www-users.cs.umn.edu/~karypis/parbook/Lectures/AG/chap4_slides.pdf

Zulfi.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/9dc3c387/attachment.html>

From knepley at gmail.com  Sat Apr  6 19:53:59 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 6 Apr 2019 20:53:59 -0400
Subject: [petsc-users] All-to-All Personalized Communication on a Mesh
In-Reply-To: <1413664627.748377.1554598090343@mail.yahoo.com>
References: <1413664627.748377.1554598090343.ref@mail.yahoo.com>
	<1413664627.748377.1554598090343@mail.yahoo.com>
Message-ID: <CAMYG4GnREEhENtwwLYVD22+=UWctv1M_-qhF2+Lpy9h-xqQ5Uw@mail.gmail.com>

On Sat, Apr 6, 2019 at 8:48 PM Zulfi Khan via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
> Can some body please tell me what would  be the result of
> applying All-to-All Personalized Communication on a Mesh shown on slide 54
> on the following link:
>
> https://www-users.cs.umn.edu/~karypis/parbook/Lectures/AG/chap4_slides.pdf
>

I cannot understand the question. It seems like "personalized" here means
Alltoallv()

   Matt


> Zulfi.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190406/df882790/attachment-0001.html>

From bsmith at mcs.anl.gov  Sat Apr  6 20:14:10 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 7 Apr 2019 01:14:10 +0000
Subject: [petsc-users] ASCIIRead error for multiple processors
In-Reply-To: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524228F4E3DE5D21C476BEE2D2500@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <474FECDA-8197-4217-8BF3-A64E01E6AC2B@anl.gov>


   I have added error checking and clearer documentation on PetscViewerASCIIRead() with https://bitbucket.org/petsc/petsc/pull-requests/1514/add-error-checking-if-petscviewerasciiread/diff 

   Thanks for the report

   Barry


> On Apr 4, 2019, at 3:15 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello team,
>  
> I?m trying to use PetscViewerASCIIRead() to read in a single interger/scalar value from an input file. It works for one processor. However, when running on multiple processors, I?m getting the below error:
>  
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR: Insufficient data, read only 0 < 1 items
> [1]PETSC ERROR: #1 PetscViewerASCIIRead() line 1054 in /usr/local/CLAB-2/petsc-3.6/src/sys/classes/viewer/impls/ascii/filev.c
>  
> Is there something wrong with how I?m implementing this, or ASCIIRead does not work with multiple processors?
>  
> Thanks,
> Yuyun


From mfadams at lbl.gov  Sat Apr  6 20:50:03 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 6 Apr 2019 21:50:03 -0400
Subject: [petsc-users] All-to-All Personalized Communication on a Mesh
In-Reply-To: <1413664627.748377.1554598090343@mail.yahoo.com>
References: <1413664627.748377.1554598090343.ref@mail.yahoo.com>
	<1413664627.748377.1554598090343@mail.yahoo.com>
Message-ID: <CADOhEh5er6wqMU3ok_cJQXD5Wah=aqgGVA3CR8-OdZckX6gvgg@mail.gmail.com>

On Sat, Apr 6, 2019 at 8:48 PM Zulfi Khan via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
> Can some body please tell me what would  be the result of
> applying All-to-All Personalized Communication on a Mesh shown on slide 54
> on the following link:
>
> https://www-users.cs.umn.edu/~karypis/parbook/Lectures/AG/chap4_slides.pdf
>

I think it is a matrix transpose ... am I right?

Note, though this is an intriguing question this is not an appropriate
forum. Maybe try https://cs.stackexchange.com

Mark


>
>
> Zulfi.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190406/f92620b0/attachment.html>

From eda.oktay at metu.edu.tr  Sun Apr  7 04:43:27 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Sun, 7 Apr 2019 12:43:27 +0300
Subject: [petsc-users] error: Petsc has generated inconsistent data,
 MPI_Allreduce() called in different locations (code lines) on
 different processors
In-Reply-To: <392602E7-E4F5-4B43-9079-41DEB1BB857B@anl.gov>
References: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>
	<CAMYG4Gk8_XSH9YSXR4Fmv8EPcRMYhiRHYjyo3F0Tbav6bdF9Gg@mail.gmail.com>
	<392602E7-E4F5-4B43-9079-41DEB1BB857B@anl.gov>
Message-ID: <CA+vu_XgjJohZzisDemtkd5Yopics6Mg39=r0bpmD=U6YfuKiEA@mail.gmail.com>

Dear Barry,

Thank you for answering. I am sending my code, makefile and 2 PETSc Binary
File where airfoil1_binary has odd numbered size (the one I got error) and
gr_30_30_binary has even numbered size (I got correct result). I got the
error for lines 164 and 169.

By the way, I think I have done something wrong at line 320 (I tried to
allocate an array twice) but I just need to reduce the error in 164 and 169
then I will get to reduce the error in line 320.

Thanks,

Eda


Smith, Barry F. <bsmith at mcs.anl.gov>, 6 Nis 2019 Cmt, 09:29 tarihinde ?unu
yazd?:

>
>   Eda,
>
>    Can you send us your code (and any needed data files)? We certainly
> expect PETSc to perform correctly if the size of the matrix cannot be
> divide by the number of processors. It is possible the problem is due to
> bugs either in MatStashScatterBegin_BTS() or in your code.
>
>    Thanks
>
>     Barry
>
>
> > On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hello,
> >
> > I am trying to calculate unweighted Laplacian of a matrix by using 2
> cores. If the size of matrix is in even number then my program works.
> However, when I try to use a matrix having odd number for size, I guess
> since size of the matrix cannot be divided into processors correctly, I get
> the following error:
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [1]PETSC ERROR: Petsc has generated inconsistent data
> > [1]PETSC ERROR: MPI_Allreduce() called in different locations (code
> lines) on different processors
> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: Petsc has generated inconsistent data
> > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code
> lines) on different processors
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > #1 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > #2 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [0]PETSC ERROR: #3 main() line 164 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> > [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > #5 MatAssemblyBegin() line 5251 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> > [0]PETSC ERROR: -unweighted
> > #6 main() line 169 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> > [1]PETSC ERROR: PETSc Option Table entries:
> > [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message
> -------send entire error message to petsc-maint at mcs.anl.gov----------
> > -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > [1]PETSC ERROR: -mat_partitioning_type parmetis
> > [1]PETSC ERROR: -unweighted
> > [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) -
> process 0
> > ----------------End of Error Message -------send entire error message to
> petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
> >
> >
> >  where line 164 in my main program is MatSetOption and line 169 is
> MatAssemblyBegin. I am new to MPI usage so I did not understand why
> MPI_Allreduce() causes a problem and how can I fix the problem.
> >
> > You have to call collective methods on all processes in the same order.
> This is not happening in your code. Beyond that,
> > there is no way for us to tell how this happened.
> >
> >   Thanks,
> >
> >      Matt
> >
> > Thank you,
> >
> > Eda
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/ca9f9f52/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: example.zip
Type: application/zip
Size: 47687 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/ca9f9f52/attachment-0001.zip>

From mfadams at lbl.gov  Sun Apr  7 08:48:24 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Sun, 7 Apr 2019 09:48:24 -0400
Subject: [petsc-users] error: Petsc has generated inconsistent data,
 MPI_Allreduce() called in different locations (code lines) on
 different processors
In-Reply-To: <CA+vu_XgjJohZzisDemtkd5Yopics6Mg39=r0bpmD=U6YfuKiEA@mail.gmail.com>
References: <CA+vu_Xi6dn+PsAMYHE4bL8CNWJyto_u7VSerRpERu05i+rVTTQ@mail.gmail.com>
	<CAMYG4Gk8_XSH9YSXR4Fmv8EPcRMYhiRHYjyo3F0Tbav6bdF9Gg@mail.gmail.com>
	<392602E7-E4F5-4B43-9079-41DEB1BB857B@anl.gov>
	<CA+vu_XgjJohZzisDemtkd5Yopics6Mg39=r0bpmD=U6YfuKiEA@mail.gmail.com>
Message-ID: <CADOhEh5KoJGKBQgkR6dN5a2Av9gh3va6Dtj_J-nGWy_pSvWEQQ@mail.gmail.com>

You have MatSetOption inside of a loop for the number of rows. This is
apparently a collective operation. Move it to line 156.

On Sun, Apr 7, 2019 at 5:44 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear Barry,
>
> Thank you for answering. I am sending my code, makefile and 2 PETSc Binary
> File where airfoil1_binary has odd numbered size (the one I got error) and
> gr_30_30_binary has even numbered size (I got correct result). I got the
> error for lines 164 and 169.
>
> By the way, I think I have done something wrong at line 320 (I tried to
> allocate an array twice) but I just need to reduce the error in 164 and 169
> then I will get to reduce the error in line 320.
>
> Thanks,
>
> Eda
>
>
> Smith, Barry F. <bsmith at mcs.anl.gov>, 6 Nis 2019 Cmt, 09:29 tarihinde
> ?unu yazd?:
>
>>
>>   Eda,
>>
>>    Can you send us your code (and any needed data files)? We certainly
>> expect PETSc to perform correctly if the size of the matrix cannot be
>> divide by the number of processors. It is possible the problem is due to
>> bugs either in MatStashScatterBegin_BTS() or in your code.
>>
>>    Thanks
>>
>>     Barry
>>
>>
>> > On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > Hello,
>> >
>> > I am trying to calculate unweighted Laplacian of a matrix by using 2
>> cores. If the size of matrix is in even number then my program works.
>> However, when I try to use a matrix having odd number for size, I guess
>> since size of the matrix cannot be divided into processors correctly, I get
>> the following error:
>> >
>> > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [1]PETSC ERROR: Petsc has generated inconsistent data
>> > [1]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: Petsc has generated inconsistent data
>> > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #1 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #2 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [0]PETSC ERROR: #3 main() line 164 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
>> > [0]PETSC ERROR: -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > #5 MatAssemblyBegin() line 5251 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
>> > [0]PETSC ERROR: -unweighted
>> > #6 main() line 169 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: PETSc Option Table entries:
>> > [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message
>> -------send entire error message to petsc-maint at mcs.anl.gov----------
>> > -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > [1]PETSC ERROR: -mat_partitioning_type parmetis
>> > [1]PETSC ERROR: -unweighted
>> > [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) -
>> process 0
>> > ----------------End of Error Message -------send entire error message
>> to petsc-maint at mcs.anl.gov----------
>> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>> >
>> >
>> >  where line 164 in my main program is MatSetOption and line 169 is
>> MatAssemblyBegin. I am new to MPI usage so I did not understand why
>> MPI_Allreduce() causes a problem and how can I fix the problem.
>> >
>> > You have to call collective methods on all processes in the same order.
>> This is not happening in your code. Beyond that,
>> > there is no way for us to tell how this happened.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > Thank you,
>> >
>> > Eda
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/686c5952/attachment.html>

From fdkong.jd at gmail.com  Sun Apr  7 18:55:53 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Sun, 7 Apr 2019 17:55:53 -0600
Subject: [petsc-users] PetscSortIntWithDataArray
Message-ID: <CAN5Wd-++XCqWZMWUv=fDG4raDKS7S6=HZViijw2c4xRrtZNm-g@mail.gmail.com>

Hi All,

*Input Parameters*

*n - number of values*
*i - array of integers*
*Ii - second array of data*
*size - sizeof elements in the data array in bytes*
*work - workspace of "size" bytes used when sorting*


size is the size of one element or all elements? It looks like it is the
size of one element when looked into the code.

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/a8688c34/attachment.html>

From jed at jedbrown.org  Sun Apr  7 19:44:25 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 07 Apr 2019 18:44:25 -0600
Subject: [petsc-users] PetscSortIntWithDataArray
In-Reply-To: <CAN5Wd-++XCqWZMWUv=fDG4raDKS7S6=HZViijw2c4xRrtZNm-g@mail.gmail.com>
References: <CAN5Wd-++XCqWZMWUv=fDG4raDKS7S6=HZViijw2c4xRrtZNm-g@mail.gmail.com>
Message-ID: <87lg0lqqae.fsf@jedbrown.org>

Fande Kong via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi All,
>
> *Input Parameters*
>
> *n - number of values*
> *i - array of integers*
> *Ii - second array of data*
> *size - sizeof elements in the data array in bytes*
> *work - workspace of "size" bytes used when sorting*
>
>
> size is the size of one element or all elements? It looks like it is the
> size of one element when looked into the code.

Each element.

From eda.oktay at metu.edu.tr  Mon Apr  8 03:21:21 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Mon, 8 Apr 2019 11:21:21 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
Message-ID: <CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>

Hello again,

I solved the problem for even numbered sized matrices. However when the
matrix size is odd, then number of elements in each index set at each
processor are different. (For example, for size 4253*4253 and 2 processors,
size of index set at processor 0 is 2127 where at processor 1, it is 2126)
I think this is why, MatPermute again gives the same "Argument out of
range" error. Index sets look like correct but I still did not get why I
get this error.

This is the part of my program:

  PetscMPIInt rank,size;
  MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
  MPI_Comm_size(PETSC_COMM_WORLD, &size);

  PetscInt mod;
  mod = siz % size;

  PetscInt *idxx,ss;
  ss = (siz-mod)/size;


  if (mod != 0){
    if (rank<mod){
        PetscMalloc1(ss+1,&idxx);
    } else{
        PetscMalloc1(ss,&idxx);
    }
  }

  if (rank != size-1) {
    j =0;
    for (i=rank*ss; i<(rank+1)*ss; i++) {
      idxx[j] = idx[i];
      j++;
    }

  } else {

    j =0;
    for (i=rank*ss; i<siz; i++) {
      idxx[j] = idx[i];
      j++;
    }

  }

  if (mod != 0){
    if (rank<mod){
        idxx[ss+1] = idx[ss*size+rank+1];
    }
  }

  /*Permute matrix L (spy(A(p1,p1))*/

    if (mod != 0){
    if (rank<mod){
        ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
    } else{
        ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
    }
  }
  ierr = ISSetPermutation(is);CHKERRQ(ierr);

  ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);

And I get the following error even if I use MatSetOption :

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
11:10:59 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
/home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #2 MatSetValues() line 1349 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
/home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #4 MatPermute() line 4997 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #5 main() line 352 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -f
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
[0]PETSC ERROR: -mat_partitioning_type parmetis
[0]PETSC ERROR: -unweighted
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------

Thanks!

Eda

Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde ?unu
yazd?:

> I attached whole program I wrote where the problem is in line 285. One of
> the matrices I used was airfoil1_binary, included in the folder. Also, I
> included makefile. Is that what you want?
>
> Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde
> ?unu yazd?:
>
>> That should not happen. Can you send in a small example that we can debug.
>>
>>   Thanks,
>>
>>     Matt
>>
>> On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hello,
>>>
>>> I am trying to permute a vector A using following lines:
>>>
>>> ierr =
>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>   ierr =
>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>
>>> However, in MatPermute line, I get the following error even if I used
>>> MatSetOption before this line:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>>> off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>> [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>> --download-metis --download-parmetis --download-superlu_dist
>>> --download-slepc --download-mpich
>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #6 main() line 285 in
>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -f
>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>> [0]PETSC ERROR: -mat_partitioning_type parmetis
>>> [0]PETSC ERROR: -weighted
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>>
>>> I'll be glad if you can help me.
>>>
>>> Thanks!
>>>
>>> Eda
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/2615b428/attachment.html>

From mfadams at lbl.gov  Mon Apr  8 08:46:36 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 8 Apr 2019 09:46:36 -0400
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
Message-ID: <CADOhEh5P=LHM9OvrfJBRtuMfqG8hOf5_mcBwCYY6L7qjckNiXg@mail.gmail.com>

Note, it does not look like "is" gets created and idxx gets allocated if
mod==0.

Are you sure 'idx' is a valid permutation? You might try replacing idx[i]
with i for debugging, and test with mod==0

On Mon, Apr 8, 2019 at 4:21 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello again,
>
> I solved the problem for even numbered sized matrices. However when the
> matrix size is odd, then number of elements in each index set at each
> processor are different. (For example, for size 4253*4253 and 2 processors,
> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
> I think this is why, MatPermute again gives the same "Argument out of
> range" error. Index sets look like correct but I still did not get why I
> get this error.
>
> This is the part of my program:
>
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>
>   PetscInt mod;
>   mod = siz % size;
>
>   PetscInt *idxx,ss;
>   ss = (siz-mod)/size;
>
>
>   if (mod != 0){
>     if (rank<mod){
>         PetscMalloc1(ss+1,&idxx);
>     } else{
>         PetscMalloc1(ss,&idxx);
>     }
>   }
>
>   if (rank != size-1) {
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>
>   } else {
>
>     j =0;
>     for (i=rank*ss; i<siz; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>
>   }
>
>   if (mod != 0){
>     if (rank<mod){
>         idxx[ss+1] = idx[ss*size+rank+1];
>     }
>   }
>
>   /*Permute matrix L (spy(A(p1,p1))*/
>
>     if (mod != 0){
>     if (rank<mod){
>         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     } else{
>         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     }
>   }
>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>
>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>
> And I get the following error even if I use MatSetOption :
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
> 11:10:59 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1349 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #4 MatPermute() line 4997 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #5 main() line 352 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
>
> Thanks!
>
> Eda
>
> Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde ?unu
> yazd?:
>
>> I attached whole program I wrote where the problem is in line 285. One of
>> the matrices I used was airfoil1_binary, included in the folder. Also, I
>> included makefile. Is that what you want?
>>
>> Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde
>> ?unu yazd?:
>>
>>> That should not happen. Can you send in a small example that we can
>>> debug.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>> On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am trying to permute a vector A using following lines:
>>>>
>>>> ierr =
>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>   ierr =
>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>
>>>> However, in MatPermute line, I get the following error even if I used
>>>> MatSetOption before this line:
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: Argument out of range
>>>> [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>> turn off this check
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #6 main() line 285 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>> [0]PETSC ERROR: PETSc Option Table entries:
>>>> [0]PETSC ERROR: -f
>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>> [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>> [0]PETSC ERROR: -weighted
>>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>>> error message to petsc-maint at mcs.anl.gov----------
>>>>
>>>> I'll be glad if you can help me.
>>>>
>>>> Thanks!
>>>>
>>>> Eda
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/4e2f89db/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Apr  8 12:12:06 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 8 Apr 2019 12:12:06 -0500
Subject: [petsc-users] Error with parallel solve
Message-ID: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>


Hi,
  
    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 

    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 

    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 

Regards,
Manav

[8]PETSC ERROR: ------------------------------------------------------------------------
[8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[8]PETSC ERROR: to get more information on the crash.
[8]PETSC ERROR: User provided function() line 0 in  unknown file  
PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
           0 :INTERNAL Error: recvd root arrowhead 
           0 :not belonging to me. IARR,JARR=       67525       67525
           0 :IROW_GRID,JCOL_GRID=           0           4
           0 :MYROW, MYCOL=           0           0
           0 :IPOSROOT,JPOSROOT=    92264688    92264688
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -99.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/25b954eb/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr  8 12:36:53 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 8 Apr 2019 17:36:53 +0000
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
Message-ID: <B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>

  Difficult to tell what is going on. 

  The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 

   Yes, debug with the debug version and also try valgrind.

   Barry


> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> Hi,
>   
>     I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> 
>     I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> 
>     I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> 
> Regards,
> Manav
> 
> [8]PETSC ERROR: ------------------------------------------------------------------------
> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [8]PETSC ERROR: to get more information on the crash.
> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
>            0 :INTERNAL Error: recvd root arrowhead 
>            0 :not belonging to me. IARR,JARR=       67525       67525
>            0 :IROW_GRID,JCOL_GRID=           0           4
>            0 :MYROW, MYCOL=           0           0
>            0 :IPOSROOT,JPOSROOT=    92264688    92264688
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode -99.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> 


From mfadams at lbl.gov  Mon Apr  8 12:58:55 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 8 Apr 2019 13:58:55 -0400
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
Message-ID: <CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>

This looks like an error in MUMPS:

        IF ( IROW_GRID .NE. root%MYROW .OR.
     &       JCOL_GRID .NE. root%MYCOL ) THEN
            WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '


On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>   Difficult to tell what is going on.
>
>   The message User provided function() line 0 in  unknown file  indicates
> the crash took place OUTSIDE of PETSc code and error message INTERNAL
> Error: recvd root arrowhead  is definitely not coming from PETSc.
>
>    Yes, debug with the debug version and also try valgrind.
>
>    Barry
>
>
> > On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> >
> > Hi,
> >
> >     I am running a code a nonlinear simulation using mesh-refinement on
> libMesh. The code runs without issues on a Mac (can run for days without
> issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux
> with openmpi 3.1.3 and gcc8.2.
> >
> >     I tried to use the -on_error_attach_debugger, but it only gave me
> this message. Does this message imply something to the more experienced
> eyes?
> >
> >     I am going to try to build a debug version of petsc to figure out
> what is going wrong. I will get and share more detailed logs in a bit.
> >
> > Regards,
> > Manav
> >
> > [8]PETSC ERROR:
> ------------------------------------------------------------------------
> > [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> > [8]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > [8]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> OS X to find memory corruption errors
> > [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> > [8]PETSC ERROR: to get more information on the crash.
> > [8]PETSC ERROR: User provided function() line 0 in  unknown file
> > PETSC: Attaching gdb to
> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > PETSC: Attaching gdb to
> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> >            0 :INTERNAL Error: recvd root arrowhead
> >            0 :not belonging to me. IARR,JARR=       67525       67525
> >            0 :IROW_GRID,JCOL_GRID=           0           4
> >            0 :MYROW, MYCOL=           0           0
> >            0 :IPOSROOT,JPOSROOT=    92264688    92264688
> >
> --------------------------------------------------------------------------
> > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > with errorcode -99.
> >
> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > You may or may not see output from other processes, depending on
> > exactly when Open MPI kills them.
> >
> --------------------------------------------------------------------------
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/d1a86834/attachment.html>

From bhatiamanav at gmail.com  Mon Apr  8 13:23:14 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 8 Apr 2019 13:23:14 -0500
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
Message-ID: <E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>

Thanks for identifying this, Mark. 

If I compile the debug version of Petsc, will it also build a debug version of Mumps? 

> On Apr 8, 2019, at 12:58 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> This looks like an error in MUMPS:
> 
>         IF ( IROW_GRID .NE. root%MYROW .OR.
>      &       JCOL_GRID .NE. root%MYCOL ) THEN
>             WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
> 
> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>   Difficult to tell what is going on. 
> 
>   The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
> 
>    Yes, debug with the debug version and also try valgrind.
> 
>    Barry
> 
> 
> > On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > 
> > 
> > Hi,
> >   
> >     I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > 
> >     I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > 
> >     I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > 
> > Regards,
> > Manav
> > 
> > [8]PETSC ERROR: ------------------------------------------------------------------------
> > [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
> > [8]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > [8]PETSC ERROR: to get more information on the crash.
> > [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> > PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> >            0 :INTERNAL Error: recvd root arrowhead 
> >            0 :not belonging to me. IARR,JARR=       67525       67525
> >            0 :IROW_GRID,JCOL_GRID=           0           4
> >            0 :MYROW, MYCOL=           0           0
> >            0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > --------------------------------------------------------------------------
> > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > with errorcode -99.
> > 
> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > You may or may not see output from other processes, depending on
> > exactly when Open MPI kills them.
> > --------------------------------------------------------------------------
> > 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/04ea668f/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Apr  8 13:23:58 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 8 Apr 2019 13:23:58 -0500
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
Message-ID: <5F685A21-7233-4ADD-9D48-1BECD703F53E@gmail.com>

BTW,  yes, I am using Mumps for the parallel direct solve. 

> On Apr 8, 2019, at 1:23 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Thanks for identifying this, Mark. 
> 
> If I compile the debug version of Petsc, will it also build a debug version of Mumps? 
> 
>> On Apr 8, 2019, at 12:58 PM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> This looks like an error in MUMPS:
>> 
>>         IF ( IROW_GRID .NE. root%MYROW .OR.
>>      &       JCOL_GRID .NE. root%MYCOL ) THEN
>>             WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
>> 
>> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>   Difficult to tell what is going on. 
>> 
>>   The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
>> 
>>    Yes, debug with the debug version and also try valgrind.
>> 
>>    Barry
>> 
>> 
>> > On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> > 
>> > 
>> > Hi,
>> >   
>> >     I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
>> > 
>> >     I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
>> > 
>> >     I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
>> > 
>> > Regards,
>> > Manav
>> > 
>> > [8]PETSC ERROR: ------------------------------------------------------------------------
>> > [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> > [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> > [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>> > [8]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
>> > [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>> > [8]PETSC ERROR: to get more information on the crash.
>> > [8]PETSC ERROR: User provided function() line 0 in  unknown file  
>> > PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
>> > PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
>> >            0 :INTERNAL Error: recvd root arrowhead 
>> >            0 :not belonging to me. IARR,JARR=       67525       67525
>> >            0 :IROW_GRID,JCOL_GRID=           0           4
>> >            0 :MYROW, MYCOL=           0           0
>> >            0 :IPOSROOT,JPOSROOT=    92264688    92264688
>> > --------------------------------------------------------------------------
>> > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> > with errorcode -99.
>> > 
>> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> > You may or may not see output from other processes, depending on
>> > exactly when Open MPI kills them.
>> > --------------------------------------------------------------------------
>> > 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/5790fdb0/attachment.html>

From mfadams at lbl.gov  Mon Apr  8 13:33:40 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 8 Apr 2019 14:33:40 -0400
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
Message-ID: <CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>

On Mon, Apr 8, 2019 at 2:23 PM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Thanks for identifying this, Mark.
>
> If I compile the debug version of Petsc, will it also build a debug
> version of Mumps?
>

The debug compiler flags will get passed down to MUMPS if you are
downloading MUMPS in PETSc. Otherwise, yes build a debug version.

Are you able to run the exact same job on your Mac? ie, same number of
processes, etc.


>
> On Apr 8, 2019, at 12:58 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
> This looks like an error in MUMPS:
>
>         IF ( IROW_GRID .NE. root%MYROW .OR.
>      &       JCOL_GRID .NE. root%MYCOL ) THEN
>             WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
>
>
> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>   Difficult to tell what is going on.
>>
>>   The message User provided function() line 0 in  unknown file  indicates
>> the crash took place OUTSIDE of PETSc code and error message INTERNAL
>> Error: recvd root arrowhead  is definitely not coming from PETSc.
>>
>>    Yes, debug with the debug version and also try valgrind.
>>
>>    Barry
>>
>>
>> > On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> >
>> > Hi,
>> >
>> >     I am running a code a nonlinear simulation using mesh-refinement on
>> libMesh. The code runs without issues on a Mac (can run for days without
>> issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux
>> with openmpi 3.1.3 and gcc8.2.
>> >
>> >     I tried to use the -on_error_attach_debugger, but it only gave me
>> this message. Does this message imply something to the more experienced
>> eyes?
>> >
>> >     I am going to try to build a debug version of petsc to figure out
>> what is going wrong. I will get and share more detailed logs in a bit.
>> >
>> > Regards,
>> > Manav
>> >
>> > [8]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> > [8]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > [8]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> > [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>> OS X to find memory corruption errors
>> > [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> > [8]PETSC ERROR: to get more information on the crash.
>> > [8]PETSC ERROR: User provided function() line 0 in  unknown file
>> > PETSC: Attaching gdb to
>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
>> of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
>> <http://warhawk1.hpc.msstate.edu/>
>> > PETSC: Attaching gdb to
>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
>> of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
>> <http://warhawk1.hpc.msstate.edu/>
>> >            0 :INTERNAL Error: recvd root arrowhead
>> >            0 :not belonging to me. IARR,JARR=       67525       67525
>> >            0 :IROW_GRID,JCOL_GRID=           0           4
>> >            0 :MYROW, MYCOL=           0           0
>> >            0 :IPOSROOT,JPOSROOT=    92264688    92264688
>> >
>> --------------------------------------------------------------------------
>> > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> > with errorcode -99.
>> >
>> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> > You may or may not see output from other processes, depending on
>> > exactly when Open MPI kills them.
>> >
>> --------------------------------------------------------------------------
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/a4e5038c/attachment.html>

From bhatiamanav at gmail.com  Mon Apr  8 13:43:46 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 8 Apr 2019 13:43:46 -0500
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
	<CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>
Message-ID: <F2476172-2DEA-44F4-9DC0-F25145043E57@gmail.com>



> On Apr 8, 2019, at 1:33 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> Are you able to run the exact same job on your Mac? ie, same number of processes, etc.

This is what I am trying to dig into now. 

My Mac has 4 cores. 

I have used several different Linux machines with different number of processors: 4, 12, 10, 20. They all eventually crash. 

I am trying to establish if the point of crash is the same across machines. 

-Manav



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/95b1aac9/attachment.html>

From balay at mcs.anl.gov  Mon Apr  8 14:19:51 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 8 Apr 2019 19:19:51 +0000
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <alpine.LFD.2.21.1904081405130.2709@sb>
References: <mailman.11368.1554747858.88853.petsc-users@mcs.anl.gov>
	<BB081502-71D6-4962-B232-26E918FB7280@ic.ac.uk>
	<8FBE599E-DE07-484C-A558-197916DA58A2@gmail.com>
	<alpine.LFD.2.21.1904081405130.2709@sb>
Message-ID: <alpine.LFD.2.21.1904081417240.2709@sb>

spack update to use mumps-5.1.2 - with this patch is in branch 'balay/mumps-5.1.2'

Satish

On Mon, 8 Apr 2019, Satish Balay wrote:

> Yes - mumps via spack is unlikely to have this patch - but it can be added.
> 
> https://bitbucket.org/petsc/pkg-mumps/commits/5fe5b9e56f78de2b7b1c199688f6c73ff3ff4c2d
> 
> Satish
> 
> On Mon, 8 Apr 2019, Manav Bhatia wrote:
> 
> > This is helpful, Thibaut. Thanks! 
> > 
> > For reference: all my Linux installs are using Spack, while my Mac install is through a petsc config where I let it download and install mumps. 
> > 
> > Could this be a source of difference in patch level for Mumps? 
> > 
> > 
> > > On Apr 8, 2019, at 1:56 PM, Appel, Thibaut <t.appel17 at imperial.ac.uk> wrote:
> > > 
> > > Hi Manav,
> > > This seems to be the bug in MUMPS that I reported to their developers last summer.
> > > But I thought Satish Balay had issued a patch in the maint branch of PETSc to correct that a few months ago?
> > > The temporary workaround was to disable the ScaLAPACK root node, ICNTL(13)=1
> > > One of the developers said later
> > >> A workaround consists in modifying the file src/dtype3_root.F near line 808
> > >> and replace the lines:
> > >> 
> > >>       SUBROUTINE DMUMPS_INIT_ROOT_FAC( N, root, FILS, IROOT,
> > >>      &                                 KEEP, INFO )
> > >>       IMPLICIT NONE
> > >>       INCLUDE 'dmumps_root.h'
> > >> by:
> > >> 
> > >>       SUBROUTINE DMUMPS_INIT_ROOT_FAC( N, root, FILS, IROOT,
> > >>      &                                 KEEP, INFO )
> > >>       USE DMUMPS_STRUC_DEF
> > >>       IMPLICIT NONE
> > >> 
> > > 
> > > Weird that you?re getting this now if it has been corrected in PETSc?
> > > 
> > > Thibaut
> > >> 
> > >> > On Apr 8, 2019, at 1:33 PM, Mark Adams <mfadams at lbl.gov <https://lists.mcs.anl.gov/mailman/listinfo/petsc-users>> wrote:
> > >> > 
> > >> > Are you able to run the exact same job on your Mac? ie, same number of processes, etc.
> > >> 
> > >> This is what I am trying to dig into now. 
> > >> 
> > >> My Mac has 4 cores. 
> > >> 
> > >> I have used several different Linux machines with different number of processors: 4, 12, 10, 20. They all eventually crash. 
> > >> 
> > >> I am trying to establish if the point of crash is the same across machines. 
> > >> 
> > >> -Manav
> > > 
> > > 
> > >> On 8 Apr 2019, at 20:24, petsc-users-request at mcs.anl.gov <mailto:petsc-users-request at mcs.anl.gov> wrote:
> > >> 
> > >> Send petsc-users mailing list submissions to
> > >> petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> > >> 
> > >> To subscribe or unsubscribe via the World Wide Web, visit
> > >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> > >> or, via email, send a message with subject or body 'help' to
> > >> petsc-users-request at mcs.anl.gov
> > >> 
> > >> You can reach the person managing the list at
> > >> petsc-users-owner at mcs.anl.gov
> > >> 
> > >> When replying, please edit your Subject line so it is more specific
> > >> than "Re: Contents of petsc-users digest..."
> > >> 
> > >> 
> > >> Today's Topics:
> > >> 
> > >>   1.  Error with parallel solve (Manav Bhatia)
> > >>   2. Re:  Error with parallel solve (Smith, Barry F.)
> > >>   3. Re:  Error with parallel solve (Mark Adams)
> > >>   4. Re:  Error with parallel solve (Manav Bhatia)
> > >> 
> > >> 
> > >> ----------------------------------------------------------------------
> > >> 
> > >> Message: 1
> > >> Date: Mon, 8 Apr 2019 12:12:06 -0500
> > >> From: Manav Bhatia <bhatiamanav at gmail.com>
> > >> To: Evan Um via petsc-users <petsc-users at mcs.anl.gov>
> > >> Subject: [petsc-users] Error with parallel solve
> > >> Message-ID: <BB21322F-A7C6-4D93-98B4-4B2D0D484724 at gmail.com>
> > >> Content-Type: text/plain; charset="us-ascii"
> > >> 
> > >> 
> > >> Hi,
> > >> 
> > >>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > >> 
> > >>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > >> 
> > >>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > >> 
> > >> Regards,
> > >> Manav
> > >> 
> > >> [8]PETSC ERROR: ------------------------------------------------------------------------
> > >> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > >> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > >> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > >> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > >> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > >> [8]PETSC ERROR: to get more information on the crash.
> > >> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > >> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >>           0 :INTERNAL Error: recvd root arrowhead 
> > >>           0 :not belonging to me. IARR,JARR=       67525       67525
> > >>           0 :IROW_GRID,JCOL_GRID=           0           4
> > >>           0 :MYROW, MYCOL=           0           0
> > >>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > >> --------------------------------------------------------------------------
> > >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > >> with errorcode -99.
> > >> 
> > >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >> You may or may not see output from other processes, depending on
> > >> exactly when Open MPI kills them.
> > >> --------------------------------------------------------------------------
> > >> 
> > >> -------------- next part --------------
> > >> An HTML attachment was scrubbed...
> > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/25b954eb/attachment-0001.html>
> > >> 
> > >> ------------------------------
> > >> 
> > >> Message: 2
> > >> Date: Mon, 8 Apr 2019 17:36:53 +0000
> > >> From: "Smith, Barry F." <bsmith at mcs.anl.gov>
> > >> To: Manav Bhatia <bhatiamanav at gmail.com>
> > >> Cc: Evan Um via petsc-users <petsc-users at mcs.anl.gov>
> > >> Subject: Re: [petsc-users] Error with parallel solve
> > >> Message-ID: <B0E28830-AAF4-425B-8C7D-63686AF1B503 at anl.gov>
> > >> Content-Type: text/plain; charset="us-ascii"
> > >> 
> > >>  Difficult to tell what is going on. 
> > >> 
> > >>  The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
> > >> 
> > >>   Yes, debug with the debug version and also try valgrind.
> > >> 
> > >>   Barry
> > >> 
> > >> 
> > >>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > >>> 
> > >>> 
> > >>> Hi,
> > >>> 
> > >>>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > >>> 
> > >>>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > >>> 
> > >>>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > >>> 
> > >>> Regards,
> > >>> Manav
> > >>> 
> > >>> [8]PETSC ERROR: ------------------------------------------------------------------------
> > >>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > >>> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > >>> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > >>> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > >>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > >>> [8]PETSC ERROR: to get more information on the crash.
> > >>> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > >>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >>>           0 :INTERNAL Error: recvd root arrowhead 
> > >>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > >>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > >>>           0 :MYROW, MYCOL=           0           0
> > >>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > >>> --------------------------------------------------------------------------
> > >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > >>> with errorcode -99.
> > >>> 
> > >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >>> You may or may not see output from other processes, depending on
> > >>> exactly when Open MPI kills them.
> > >>> --------------------------------------------------------------------------
> > >>> 
> > >> 
> > >> 
> > >> 
> > >> ------------------------------
> > >> 
> > >> Message: 3
> > >> Date: Mon, 8 Apr 2019 13:58:55 -0400
> > >> From: Mark Adams <mfadams at lbl.gov>
> > >> To: "Smith, Barry F." <bsmith at mcs.anl.gov>
> > >> Cc: Manav Bhatia <bhatiamanav at gmail.com>, 
> > >> Evan Um via petsc-users
> > >> <petsc-users at mcs.anl.gov>
> > >> Subject: Re: [petsc-users] Error with parallel solve
> > >> Message-ID:
> > >> <CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA at mail.gmail.com>
> > >> Content-Type: text/plain; charset="utf-8"
> > >> 
> > >> This looks like an error in MUMPS:
> > >> 
> > >>        IF ( IROW_GRID .NE. root%MYROW .OR.
> > >>     &       JCOL_GRID .NE. root%MYCOL ) THEN
> > >>            WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
> > >> 
> > >> 
> > >> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <
> > >> petsc-users at mcs.anl.gov> wrote:
> > >> 
> > >>>  Difficult to tell what is going on.
> > >>> 
> > >>>  The message User provided function() line 0 in  unknown file  indicates
> > >>> the crash took place OUTSIDE of PETSc code and error message INTERNAL
> > >>> Error: recvd root arrowhead  is definitely not coming from PETSc.
> > >>> 
> > >>>   Yes, debug with the debug version and also try valgrind.
> > >>> 
> > >>>   Barry
> > >>> 
> > >>> 
> > >>>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <
> > >>> petsc-users at mcs.anl.gov> wrote:
> > >>>> 
> > >>>> 
> > >>>> Hi,
> > >>>> 
> > >>>>    I am running a code a nonlinear simulation using mesh-refinement on
> > >>> libMesh. The code runs without issues on a Mac (can run for days without
> > >>> issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux
> > >>> with openmpi 3.1.3 and gcc8.2.
> > >>>> 
> > >>>>    I tried to use the -on_error_attach_debugger, but it only gave me
> > >>> this message. Does this message imply something to the more experienced
> > >>> eyes?
> > >>>> 
> > >>>>    I am going to try to build a debug version of petsc to figure out
> > >>> what is going wrong. I will get and share more detailed logs in a bit.
> > >>>> 
> > >>>> Regards,
> > >>>> Manav
> > >>>> 
> > >>>> [8]PETSC ERROR:
> > >>> ------------------------------------------------------------------------
> > >>>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > >>> probably memory access out of range
> > >>>> [8]PETSC ERROR: Try option -start_in_debugger or
> > >>> -on_error_attach_debugger
> > >>>> [8]PETSC ERROR: or see
> > >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > >>>> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> > >>> OS X to find memory corruption errors
> > >>>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > >>> and run
> > >>>> [8]PETSC ERROR: to get more information on the crash.
> > >>>> [8]PETSC ERROR: User provided function() line 0 in  unknown file
> > >>>> PETSC: Attaching gdb to
> > >>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> > >>> of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >>>> PETSC: Attaching gdb to
> > >>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> > >>> of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > >>>>           0 :INTERNAL Error: recvd root arrowhead
> > >>>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > >>>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > >>>>           0 :MYROW, MYCOL=           0           0
> > >>>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > >>>> 
> > >>> --------------------------------------------------------------------------
> > >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > >>>> with errorcode -99.
> > >>>> 
> > >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >>>> You may or may not see output from other processes, depending on
> > >>>> exactly when Open MPI kills them.
> > >>>> 
> > >>> --------------------------------------------------------------------------
> > >>>> 
> > >>> 
> > >>> 
> > >> -------------- next part --------------
> > >> An HTML attachment was scrubbed...
> > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/d1a86834/attachment-0001.html>
> > >> 
> > >> ------------------------------
> > >> 
> > >> Message: 4
> > >> Date: Mon, 8 Apr 2019 13:23:14 -0500
> > >> From: Manav Bhatia <bhatiamanav at gmail.com>
> > >> To: Mark Adams <mfadams at lbl.gov>
> > >> Cc: "Smith, Barry F." <bsmith at mcs.anl.gov>,
> > >> Evan Um via petsc-users
> > >> <petsc-users at mcs.anl.gov>
> > >> Subject: Re: [petsc-users] Error with parallel solve
> > >> Message-ID: <E1168C22-17F5-4A5F-870A-88F0D5FBFA31 at gmail.com>
> > >> Content-Type: text/plain; charset="us-ascii"
> > >> 
> > >> Thanks for identifying this, Mark. 
> > >> 
> > >> If I compile the debug version of Petsc, will it also build a debug version of Mumps? 
> > >> 
> > >>> On Apr 8, 2019, at 12:58 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > >>> 
> > >>> This looks like an error in MUMPS:
> > >>> 
> > >>>        IF ( IROW_GRID .NE. root%MYROW .OR.
> > >>>     &       JCOL_GRID .NE. root%MYCOL ) THEN
> > >>>            WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
> > >>> 
> > >>> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > >>>  Difficult to tell what is going on. 
> > >>> 
> > >>>  The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
> > >>> 
> > >>>   Yes, debug with the debug version and also try valgrind.
> > >>> 
> > >>>   Barry
> > >>> 
> > >>> 
> > >>>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > >>>> 
> > >>>> 
> > >>>> Hi,
> > >>>> 
> > >>>>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > >>>> 
> > >>>>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > >>>> 
> > >>>>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > >>>> 
> > >>>> Regards,
> > >>>> Manav
> > >>>> 
> > >>>> [8]PETSC ERROR: ------------------------------------------------------------------------
> > >>>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > >>>> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > >>>> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
> > >>>> [8]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
> > >>>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > >>>> [8]PETSC ERROR: to get more information on the crash.
> > >>>> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > >>>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> > >>>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> > >>>>           0 :INTERNAL Error: recvd root arrowhead 
> > >>>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > >>>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > >>>>           0 :MYROW, MYCOL=           0           0
> > >>>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > >>>> --------------------------------------------------------------------------
> > >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > >>>> with errorcode -99.
> > >>>> 
> > >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > >>>> You may or may not see output from other processes, depending on
> > >>>> exactly when Open MPI kills them.
> > >>>> --------------------------------------------------------------------------
> > >>>> 
> > >>> 
> > >> 
> > >> -------------- next part --------------
> > >> An HTML attachment was scrubbed...
> > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/04ea668f/attachment.html>
> > >> 
> > >> ------------------------------
> > >> 
> > >> Subject: Digest Footer
> > >> 
> > >> _______________________________________________
> > >> petsc-users mailing list
> > >> petsc-users at mcs.anl.gov
> > >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> > >> 
> > >> 
> > >> ------------------------------
> > >> 
> > >> End of petsc-users Digest, Vol 124, Issue 31
> > >> ********************************************
> > > 
> > 
> > 
> 

From stefano.zampini at gmail.com  Mon Apr  8 14:19:53 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 8 Apr 2019 22:19:53 +0300
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <F2476172-2DEA-44F4-9DC0-F25145043E57@gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
	<CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>
	<F2476172-2DEA-44F4-9DC0-F25145043E57@gmail.com>
Message-ID: <CAGPUisiCEyUh+sRToxFVxFW+Qu59t4v8MWqGMxRrmKCgbtM49w@mail.gmail.com>

This is a bug in the most recent version of mumps related with using
scalapack for the factorization of the root node of the nested dissection
tree. I have reported it already on this list few months ago.
You can circumvent the problem by using a sequential solver for it. There's
a command line option in petsc as well as API that allows you to do so.
-mat_mumps_icntl_13 1

Il Lun 8 Apr 2019, 21:44 Manav Bhatia via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

>
>
> On Apr 8, 2019, at 1:33 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
> Are you able to run the exact same job on your Mac? ie, same number of
> processes, etc.
>
>
> This is what I am trying to dig into now.
>
> My Mac has 4 cores.
>
> I have used several different Linux machines with different number of
> processors: 4, 12, 10, 20. They all eventually crash.
>
> I am trying to establish if the point of crash is the same across
> machines.
>
> -Manav
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/5e5d7577/attachment.html>

From bhatiamanav at gmail.com  Mon Apr  8 14:25:35 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 8 Apr 2019 14:25:35 -0500
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <CAGPUisiCEyUh+sRToxFVxFW+Qu59t4v8MWqGMxRrmKCgbtM49w@mail.gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
	<CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>
	<F2476172-2DEA-44F4-9DC0-F25145043E57@gmail.com>
	<CAGPUisiCEyUh+sRToxFVxFW+Qu59t4v8MWqGMxRrmKCgbtM49w@mail.gmail.com>
Message-ID: <B7FF0385-4339-4B83-BB48-748E1186DEB2@gmail.com>



> On Apr 8, 2019, at 2:19 PM, Stefano Zampini <stefano.zampini at gmail.com> wrote:
> 
> You can circumvent the problem by using a sequential solver for it. There's a command line option in petsc as well as API that allows you to do so. -mat_mumps_icntl_13 1

Stefano, 

  Do you know if there is a performance penalty to using this option as opposed to fixing it with the patch? 

-Manav
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/cbebbf61/attachment.html>

From balay at mcs.anl.gov  Mon Apr  8 14:38:01 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 8 Apr 2019 19:38:01 +0000
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <alpine.LFD.2.21.1904081417240.2709@sb>
References: <mailman.11368.1554747858.88853.petsc-users@mcs.anl.gov>
	<BB081502-71D6-4962-B232-26E918FB7280@ic.ac.uk>
	<8FBE599E-DE07-484C-A558-197916DA58A2@gmail.com>
	<alpine.LFD.2.21.1904081405130.2709@sb>
	<alpine.LFD.2.21.1904081417240.2709@sb>
Message-ID: <alpine.LFD.2.21.1904081437340.2709@sb>

https://github.com/spack/spack/pull/11132

If this works - please add a comment on the PR

Satish

On Mon, 8 Apr 2019, Balay, Satish via petsc-users wrote:

> spack update to use mumps-5.1.2 - with this patch is in branch 'balay/mumps-5.1.2'
> 
> Satish
> 
> On Mon, 8 Apr 2019, Satish Balay wrote:
> 
> > Yes - mumps via spack is unlikely to have this patch - but it can be added.
> > 
> > https://bitbucket.org/petsc/pkg-mumps/commits/5fe5b9e56f78de2b7b1c199688f6c73ff3ff4c2d
> > 
> > Satish
> > 
> > On Mon, 8 Apr 2019, Manav Bhatia wrote:
> > 
> > > This is helpful, Thibaut. Thanks! 
> > > 
> > > For reference: all my Linux installs are using Spack, while my Mac install is through a petsc config where I let it download and install mumps. 
> > > 
> > > Could this be a source of difference in patch level for Mumps? 
> > > 
> > > 
> > > > On Apr 8, 2019, at 1:56 PM, Appel, Thibaut <t.appel17 at imperial.ac.uk> wrote:
> > > > 
> > > > Hi Manav,
> > > > This seems to be the bug in MUMPS that I reported to their developers last summer.
> > > > But I thought Satish Balay had issued a patch in the maint branch of PETSc to correct that a few months ago?
> > > > The temporary workaround was to disable the ScaLAPACK root node, ICNTL(13)=1
> > > > One of the developers said later
> > > >> A workaround consists in modifying the file src/dtype3_root.F near line 808
> > > >> and replace the lines:
> > > >> 
> > > >>       SUBROUTINE DMUMPS_INIT_ROOT_FAC( N, root, FILS, IROOT,
> > > >>      &                                 KEEP, INFO )
> > > >>       IMPLICIT NONE
> > > >>       INCLUDE 'dmumps_root.h'
> > > >> by:
> > > >> 
> > > >>       SUBROUTINE DMUMPS_INIT_ROOT_FAC( N, root, FILS, IROOT,
> > > >>      &                                 KEEP, INFO )
> > > >>       USE DMUMPS_STRUC_DEF
> > > >>       IMPLICIT NONE
> > > >> 
> > > > 
> > > > Weird that you?re getting this now if it has been corrected in PETSc?
> > > > 
> > > > Thibaut
> > > >> 
> > > >> > On Apr 8, 2019, at 1:33 PM, Mark Adams <mfadams at lbl.gov <https://lists.mcs.anl.gov/mailman/listinfo/petsc-users>> wrote:
> > > >> > 
> > > >> > Are you able to run the exact same job on your Mac? ie, same number of processes, etc.
> > > >> 
> > > >> This is what I am trying to dig into now. 
> > > >> 
> > > >> My Mac has 4 cores. 
> > > >> 
> > > >> I have used several different Linux machines with different number of processors: 4, 12, 10, 20. They all eventually crash. 
> > > >> 
> > > >> I am trying to establish if the point of crash is the same across machines. 
> > > >> 
> > > >> -Manav
> > > > 
> > > > 
> > > >> On 8 Apr 2019, at 20:24, petsc-users-request at mcs.anl.gov <mailto:petsc-users-request at mcs.anl.gov> wrote:
> > > >> 
> > > >> Send petsc-users mailing list submissions to
> > > >> petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> > > >> 
> > > >> To subscribe or unsubscribe via the World Wide Web, visit
> > > >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> > > >> or, via email, send a message with subject or body 'help' to
> > > >> petsc-users-request at mcs.anl.gov
> > > >> 
> > > >> You can reach the person managing the list at
> > > >> petsc-users-owner at mcs.anl.gov
> > > >> 
> > > >> When replying, please edit your Subject line so it is more specific
> > > >> than "Re: Contents of petsc-users digest..."
> > > >> 
> > > >> 
> > > >> Today's Topics:
> > > >> 
> > > >>   1.  Error with parallel solve (Manav Bhatia)
> > > >>   2. Re:  Error with parallel solve (Smith, Barry F.)
> > > >>   3. Re:  Error with parallel solve (Mark Adams)
> > > >>   4. Re:  Error with parallel solve (Manav Bhatia)
> > > >> 
> > > >> 
> > > >> ----------------------------------------------------------------------
> > > >> 
> > > >> Message: 1
> > > >> Date: Mon, 8 Apr 2019 12:12:06 -0500
> > > >> From: Manav Bhatia <bhatiamanav at gmail.com>
> > > >> To: Evan Um via petsc-users <petsc-users at mcs.anl.gov>
> > > >> Subject: [petsc-users] Error with parallel solve
> > > >> Message-ID: <BB21322F-A7C6-4D93-98B4-4B2D0D484724 at gmail.com>
> > > >> Content-Type: text/plain; charset="us-ascii"
> > > >> 
> > > >> 
> > > >> Hi,
> > > >> 
> > > >>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > > >> 
> > > >>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > > >> 
> > > >>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > > >> 
> > > >> Regards,
> > > >> Manav
> > > >> 
> > > >> [8]PETSC ERROR: ------------------------------------------------------------------------
> > > >> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > >> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > >> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > >> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > >> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > > >> [8]PETSC ERROR: to get more information on the crash.
> > > >> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > > >> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >>           0 :INTERNAL Error: recvd root arrowhead 
> > > >>           0 :not belonging to me. IARR,JARR=       67525       67525
> > > >>           0 :IROW_GRID,JCOL_GRID=           0           4
> > > >>           0 :MYROW, MYCOL=           0           0
> > > >>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > > >> --------------------------------------------------------------------------
> > > >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > >> with errorcode -99.
> > > >> 
> > > >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > >> You may or may not see output from other processes, depending on
> > > >> exactly when Open MPI kills them.
> > > >> --------------------------------------------------------------------------
> > > >> 
> > > >> -------------- next part --------------
> > > >> An HTML attachment was scrubbed...
> > > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/25b954eb/attachment-0001.html>
> > > >> 
> > > >> ------------------------------
> > > >> 
> > > >> Message: 2
> > > >> Date: Mon, 8 Apr 2019 17:36:53 +0000
> > > >> From: "Smith, Barry F." <bsmith at mcs.anl.gov>
> > > >> To: Manav Bhatia <bhatiamanav at gmail.com>
> > > >> Cc: Evan Um via petsc-users <petsc-users at mcs.anl.gov>
> > > >> Subject: Re: [petsc-users] Error with parallel solve
> > > >> Message-ID: <B0E28830-AAF4-425B-8C7D-63686AF1B503 at anl.gov>
> > > >> Content-Type: text/plain; charset="us-ascii"
> > > >> 
> > > >>  Difficult to tell what is going on. 
> > > >> 
> > > >>  The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
> > > >> 
> > > >>   Yes, debug with the debug version and also try valgrind.
> > > >> 
> > > >>   Barry
> > > >> 
> > > >> 
> > > >>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > >>> 
> > > >>> 
> > > >>> Hi,
> > > >>> 
> > > >>>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > > >>> 
> > > >>>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > > >>> 
> > > >>>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > > >>> 
> > > >>> Regards,
> > > >>> Manav
> > > >>> 
> > > >>> [8]PETSC ERROR: ------------------------------------------------------------------------
> > > >>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > >>> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > >>> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > >>> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > >>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > > >>> [8]PETSC ERROR: to get more information on the crash.
> > > >>> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > > >>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >>>           0 :INTERNAL Error: recvd root arrowhead 
> > > >>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > > >>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > > >>>           0 :MYROW, MYCOL=           0           0
> > > >>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > > >>> --------------------------------------------------------------------------
> > > >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > >>> with errorcode -99.
> > > >>> 
> > > >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > >>> You may or may not see output from other processes, depending on
> > > >>> exactly when Open MPI kills them.
> > > >>> --------------------------------------------------------------------------
> > > >>> 
> > > >> 
> > > >> 
> > > >> 
> > > >> ------------------------------
> > > >> 
> > > >> Message: 3
> > > >> Date: Mon, 8 Apr 2019 13:58:55 -0400
> > > >> From: Mark Adams <mfadams at lbl.gov>
> > > >> To: "Smith, Barry F." <bsmith at mcs.anl.gov>
> > > >> Cc: Manav Bhatia <bhatiamanav at gmail.com>, 
> > > >> Evan Um via petsc-users
> > > >> <petsc-users at mcs.anl.gov>
> > > >> Subject: Re: [petsc-users] Error with parallel solve
> > > >> Message-ID:
> > > >> <CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA at mail.gmail.com>
> > > >> Content-Type: text/plain; charset="utf-8"
> > > >> 
> > > >> This looks like an error in MUMPS:
> > > >> 
> > > >>        IF ( IROW_GRID .NE. root%MYROW .OR.
> > > >>     &       JCOL_GRID .NE. root%MYCOL ) THEN
> > > >>            WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
> > > >> 
> > > >> 
> > > >> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <
> > > >> petsc-users at mcs.anl.gov> wrote:
> > > >> 
> > > >>>  Difficult to tell what is going on.
> > > >>> 
> > > >>>  The message User provided function() line 0 in  unknown file  indicates
> > > >>> the crash took place OUTSIDE of PETSc code and error message INTERNAL
> > > >>> Error: recvd root arrowhead  is definitely not coming from PETSc.
> > > >>> 
> > > >>>   Yes, debug with the debug version and also try valgrind.
> > > >>> 
> > > >>>   Barry
> > > >>> 
> > > >>> 
> > > >>>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <
> > > >>> petsc-users at mcs.anl.gov> wrote:
> > > >>>> 
> > > >>>> 
> > > >>>> Hi,
> > > >>>> 
> > > >>>>    I am running a code a nonlinear simulation using mesh-refinement on
> > > >>> libMesh. The code runs without issues on a Mac (can run for days without
> > > >>> issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux
> > > >>> with openmpi 3.1.3 and gcc8.2.
> > > >>>> 
> > > >>>>    I tried to use the -on_error_attach_debugger, but it only gave me
> > > >>> this message. Does this message imply something to the more experienced
> > > >>> eyes?
> > > >>>> 
> > > >>>>    I am going to try to build a debug version of petsc to figure out
> > > >>> what is going wrong. I will get and share more detailed logs in a bit.
> > > >>>> 
> > > >>>> Regards,
> > > >>>> Manav
> > > >>>> 
> > > >>>> [8]PETSC ERROR:
> > > >>> ------------------------------------------------------------------------
> > > >>>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > > >>> probably memory access out of range
> > > >>>> [8]PETSC ERROR: Try option -start_in_debugger or
> > > >>> -on_error_attach_debugger
> > > >>>> [8]PETSC ERROR: or see
> > > >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > >>>> [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> > > >>> OS X to find memory corruption errors
> > > >>>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > > >>> and run
> > > >>>> [8]PETSC ERROR: to get more information on the crash.
> > > >>>> [8]PETSC ERROR: User provided function() line 0 in  unknown file
> > > >>>> PETSC: Attaching gdb to
> > > >>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> > > >>> of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >>>> PETSC: Attaching gdb to
> > > >>> /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5
> > > >>> of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu
> > > >>>>           0 :INTERNAL Error: recvd root arrowhead
> > > >>>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > > >>>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > > >>>>           0 :MYROW, MYCOL=           0           0
> > > >>>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > > >>>> 
> > > >>> --------------------------------------------------------------------------
> > > >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > >>>> with errorcode -99.
> > > >>>> 
> > > >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > >>>> You may or may not see output from other processes, depending on
> > > >>>> exactly when Open MPI kills them.
> > > >>>> 
> > > >>> --------------------------------------------------------------------------
> > > >>>> 
> > > >>> 
> > > >>> 
> > > >> -------------- next part --------------
> > > >> An HTML attachment was scrubbed...
> > > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/d1a86834/attachment-0001.html>
> > > >> 
> > > >> ------------------------------
> > > >> 
> > > >> Message: 4
> > > >> Date: Mon, 8 Apr 2019 13:23:14 -0500
> > > >> From: Manav Bhatia <bhatiamanav at gmail.com>
> > > >> To: Mark Adams <mfadams at lbl.gov>
> > > >> Cc: "Smith, Barry F." <bsmith at mcs.anl.gov>,
> > > >> Evan Um via petsc-users
> > > >> <petsc-users at mcs.anl.gov>
> > > >> Subject: Re: [petsc-users] Error with parallel solve
> > > >> Message-ID: <E1168C22-17F5-4A5F-870A-88F0D5FBFA31 at gmail.com>
> > > >> Content-Type: text/plain; charset="us-ascii"
> > > >> 
> > > >> Thanks for identifying this, Mark. 
> > > >> 
> > > >> If I compile the debug version of Petsc, will it also build a debug version of Mumps? 
> > > >> 
> > > >>> On Apr 8, 2019, at 12:58 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > > >>> 
> > > >>> This looks like an error in MUMPS:
> > > >>> 
> > > >>>        IF ( IROW_GRID .NE. root%MYROW .OR.
> > > >>>     &       JCOL_GRID .NE. root%MYCOL ) THEN
> > > >>>            WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '
> > > >>> 
> > > >>> On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > > >>>  Difficult to tell what is going on. 
> > > >>> 
> > > >>>  The message User provided function() line 0 in  unknown file  indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead  is definitely not coming from PETSc. 
> > > >>> 
> > > >>>   Yes, debug with the debug version and also try valgrind.
> > > >>> 
> > > >>>   Barry
> > > >>> 
> > > >>> 
> > > >>>> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > > >>>> 
> > > >>>> 
> > > >>>> Hi,
> > > >>>> 
> > > >>>>    I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. 
> > > >>>> 
> > > >>>>    I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? 
> > > >>>> 
> > > >>>>    I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. 
> > > >>>> 
> > > >>>> Regards,
> > > >>>> Manav
> > > >>>> 
> > > >>>> [8]PETSC ERROR: ------------------------------------------------------------------------
> > > >>>> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > > >>>> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > >>>> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
> > > >>>> [8]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors
> > > >>>> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> > > >>>> [8]PETSC ERROR: to get more information on the crash.
> > > >>>> [8]PETSC ERROR: User provided function() line 0 in  unknown file  
> > > >>>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> > > >>>> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine Warhawk1.HPC.MsState.Edu <http://warhawk1.hpc.msstate.edu/>
> > > >>>>           0 :INTERNAL Error: recvd root arrowhead 
> > > >>>>           0 :not belonging to me. IARR,JARR=       67525       67525
> > > >>>>           0 :IROW_GRID,JCOL_GRID=           0           4
> > > >>>>           0 :MYROW, MYCOL=           0           0
> > > >>>>           0 :IPOSROOT,JPOSROOT=    92264688    92264688
> > > >>>> --------------------------------------------------------------------------
> > > >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > >>>> with errorcode -99.
> > > >>>> 
> > > >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > >>>> You may or may not see output from other processes, depending on
> > > >>>> exactly when Open MPI kills them.
> > > >>>> --------------------------------------------------------------------------
> > > >>>> 
> > > >>> 
> > > >> 
> > > >> -------------- next part --------------
> > > >> An HTML attachment was scrubbed...
> > > >> URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190408/04ea668f/attachment.html>
> > > >> 
> > > >> ------------------------------
> > > >> 
> > > >> Subject: Digest Footer
> > > >> 
> > > >> _______________________________________________
> > > >> petsc-users mailing list
> > > >> petsc-users at mcs.anl.gov
> > > >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> > > >> 
> > > >> 
> > > >> ------------------------------
> > > >> 
> > > >> End of petsc-users Digest, Vol 124, Issue 31
> > > >> ********************************************
> > > > 
> > > 
> > > 
> > 
> 

From balay at mcs.anl.gov  Mon Apr  8 14:39:37 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 8 Apr 2019 19:39:37 +0000
Subject: [petsc-users] Error with parallel solve
In-Reply-To: <B7FF0385-4339-4B83-BB48-748E1186DEB2@gmail.com>
References: <BB21322F-A7C6-4D93-98B4-4B2D0D484724@gmail.com>
	<B0E28830-AAF4-425B-8C7D-63686AF1B503@anl.gov>
	<CADOhEh6EZqtFkwzJYm0iCOMX5_uTfdbuH6t4WwU7_dUeY9KpzA@mail.gmail.com>
	<E1168C22-17F5-4A5F-870A-88F0D5FBFA31@gmail.com>
	<CADOhEh4QNAk+Oe0NBzbaWJxY98-1Ckmjqet1cHntoH_=GQX4gA@mail.gmail.com>
	<F2476172-2DEA-44F4-9DC0-F25145043E57@gmail.com>
	<CAGPUisiCEyUh+sRToxFVxFW+Qu59t4v8MWqGMxRrmKCgbtM49w@mail.gmail.com>
	<B7FF0385-4339-4B83-BB48-748E1186DEB2@gmail.com>
Message-ID: <alpine.LFD.2.21.1904081438100.2709@sb>

On Mon, 8 Apr 2019, Manav Bhatia via petsc-users wrote:

> 
> 
> > On Apr 8, 2019, at 2:19 PM, Stefano Zampini <stefano.zampini at gmail.com> wrote:
> > 
> > You can circumvent the problem by using a sequential solver for it. There's a command line option in petsc as well as API that allows you to do so. -mat_mumps_icntl_13 1
> 
> Stefano, 
> 
>   Do you know if there is a performance penalty to using this option as opposed to fixing it with the patch? 

If would suggest first trying both the fixes if see either of them work for you.

Satish

From bsmith at mcs.anl.gov  Mon Apr  8 16:31:44 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 8 Apr 2019 21:31:44 +0000
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
Message-ID: <DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>


  Suggest printing out the IS with ISView after it is created and confirming that 1) it is a permutation and 2) that the size of the IS on each process matches the number of rows on that process. 

  Note from the manual page: The index sets should be on the same communicator as Mat and have the same local sizes.

Barry


> On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello again,
> 
> I solved the problem for even numbered sized matrices. However when the matrix size is odd, then number of elements in each index set at each processor are different. (For example, for size 4253*4253 and 2 processors, size of index set at processor 0 is 2127 where at processor 1, it is 2126) I think this is why, MatPermute again gives the same "Argument out of range" error. Index sets look like correct but I still did not get why I get this error.
> 
> This is the part of my program:
> 
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>  
>   PetscInt mod;
>   mod = siz % size;
>   
>   PetscInt *idxx,ss;
>   ss = (siz-mod)/size;                                                        
>  
>   if (mod != 0){
>     if (rank<mod){
>         PetscMalloc1(ss+1,&idxx);                                        
>     } else{
>         PetscMalloc1(ss,&idxx);
>     }
>   }
>   
>   if (rank != size-1) {
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
> 
>   } else {
>       
>     j =0;
>     for (i=rank*ss; i<siz; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>       
>   }
>   
>   if (mod != 0){
>     if (rank<mod){                               
>         idxx[ss+1] = idx[ss*size+rank+1];
>     }
>   }
>  
>   /*Permute matrix L (spy(A(p1,p1))*/
> 
>     if (mod != 0){
>     if (rank<mod){
>         ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     } else{
>         ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     }
>   }
>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
> 
>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr); 
> 
> And I get the following error even if I use MatSetOption :
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8 11:10:59 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1349 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #4 MatPermute() line 4997 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #5 main() line 352 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 
> Thanks!
> 
> Eda
> 
> Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde ?unu yazd?:
> I attached whole program I wrote where the problem is in line 285. One of the matrices I used was airfoil1_binary, included in the folder. Also, I included makefile. Is that what you want?
> 
> Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde ?unu yazd?:
> That should not happen. Can you send in a small example that we can debug.
> 
>   Thanks,
> 
>     Matt
> 
> On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello,
> 
> I am trying to permute a vector A using following lines:  
> 
> ierr = ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>   ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr); 
>   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr); 
> 
> However, in MatPermute line, I get the following error even if I used MatSetOption before this line:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #5 MatPermute() line 4997 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #6 main() line 285 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -weighted
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 
> I'll be glad if you can help me.
> 
> Thanks!
> 
> Eda
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From ztdepyahoo at gmail.com  Tue Apr  9 03:07:04 2019
From: ztdepyahoo at gmail.com (ztdepyahoo)
Date: Tue, 9 Apr 2019 16:07:04 +0800
Subject: [petsc-users] How to decompose and record a constant A
Message-ID: <ABA46849-BEA2-456F-BB3B-368B33D8BDB8@gmail.com>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/2548fcc2/attachment.html>

From mfadams at lbl.gov  Tue Apr  9 03:33:16 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 9 Apr 2019 04:33:16 -0400
Subject: [petsc-users] How to decompose and record a constant A
In-Reply-To: <ABA46849-BEA2-456F-BB3B-368B33D8BDB8@gmail.com>
References: <ABA46849-BEA2-456F-BB3B-368B33D8BDB8@gmail.com>
Message-ID: <CADOhEh5vkK9Hw4rR+t2EPD2wHmy-XrYUKbyJRD75JN=TaF8arQ@mail.gmail.com>

PETSc solvers will cache any setup computations, like a matrix
factorization, and will reuse them on subsequent solves.

If you call KSPSetOperators, PETSc will assume its cache is invalid and
redo any required setup on subsequent solves.

Mark

On Tue, Apr 9, 2019 at 4:08 AM ztdepyahoo via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear group:
>       I need to solve a Laplacian problem Ax=b every time step where A is
> a constant . I want to store the inverse of A at the first time step.
> how to do with Petsc
>  Regards
>
> ztdepyahoo
> ztdepyahoo at gmail.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=ztdepyahoo&uid=ztdepyahoo%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22ztdepyahoo%40gmail.com%22%5D>
> ??? ?????? <https://mail.163.com/dashi/dlpro.html?from=mail81> ??
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/11330f49/attachment-0001.html>

From myriam.peyrounette at idris.fr  Tue Apr  9 07:43:00 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Tue, 9 Apr 2019 14:43:00 +0200
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <877ecaeo63.fsf@jedbrown.org>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<7b104336-a067-e679-23ec-2a89e0ba9bc4@idris.fr>
	<CADOhEh5MHFQOoHF24hOWhoAiLnqYuZO29hwKzHmMndicF1xj0g@mail.gmail.com>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
Message-ID: <9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>

Hi,

in my first mail, I provided a memory scaling concerning the PETSc
example #42. You'll find attached the main files used (one for PETSc
3.6.4, one for PETSc 3.10.2), and the corresponding memory scaling.

In the main files, I modified the solver/preconditioner, so that it
corresponds to my problem. You'll find the modifications by searching
the keyword "TopBridge". In particular, I use GAMG.

Note that the example is about solving Stokes equation, so using GAMG
may not be adapted. However, the memory gap appears and that's the
point. No matter if the results are correct.

Are these scripts useful for you? Let me know.

Thanks,

Myriam


Le 04/04/19 ? 00:09, Jed Brown a ?crit?:
> Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov> writes:
>
>> Hi all,
>>
>> for your information, you'll find attached the comparison of the weak
>> memory scalings when using :
>>
>> - PETSc 3.6.4 (reference)
>> - PETSc 3.10.4 without specific options
>> - PETSc 3.10.4 with the three scalability options you mentionned
>>
>> Using the scalability options does improve the memory scaling. However,
>> the 3.6 version still has a better one...
> Yes, this still looks significant.  Is this an effect we can still
> reproduce with a PETSc example and/or using a memory profiler (such as
> massif or gperftools)?  I think it's important for us to narrow down
> what causes this difference (looks like almost 2x on your 1e8 problem
> size) so we can fix.

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
A non-text attachment was scrubbed...
Name: scaling_ex42.png
Type: image/png
Size: 24427 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/d913744d/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main_ex42_petsc3102.cc
Type: text/x-c++src
Size: 88894 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/d913744d/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main_ex42_petsc364.cc
Type: text/x-c++src
Size: 88647 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/d913744d/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/d913744d/attachment-0001.p7s>

From dave.mayhem23 at gmail.com  Tue Apr  9 08:32:33 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 9 Apr 2019 15:32:33 +0200
Subject: [petsc-users] How to decompose and record a constant A
In-Reply-To: <CADOhEh5vkK9Hw4rR+t2EPD2wHmy-XrYUKbyJRD75JN=TaF8arQ@mail.gmail.com>
References: <ABA46849-BEA2-456F-BB3B-368B33D8BDB8@gmail.com>
	<CADOhEh5vkK9Hw4rR+t2EPD2wHmy-XrYUKbyJRD75JN=TaF8arQ@mail.gmail.com>
Message-ID: <CAJ98EDqrOkHxyh5m6=3DQxY_b=vGynWRvPqkpqd6JG+XjOvMzw@mail.gmail.com>

On Tue, 9 Apr 2019 at 10:33, Mark Adams via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> PETSc solvers will cache any setup computations, like a matrix
> factorization, and will reuse them on subsequent solves.
>
> If you call KSPSetOperators, PETSc will assume its cache is invalid and
> redo any required setup on subsequent solves.
>

Following from what Mark said, if you want an exact solve, use -pc_type lu.
Note that this will not store the inverse, rather only the L, U factors.
During setup, the L,U factors are created and subsequent solves will simply
apply the forward and backward solves using these factors.



> Mark
>
> On Tue, Apr 9, 2019 at 4:08 AM ztdepyahoo via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear group:
>>       I need to solve a Laplacian problem Ax=b every time step where A is
>> a constant . I want to store the inverse of A at the first time step.
>> how to do with Petsc
>>  Regards
>>
>> ztdepyahoo
>> ztdepyahoo at gmail.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=ztdepyahoo&uid=ztdepyahoo%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22ztdepyahoo%40gmail.com%22%5D>
>> ??? ?????? <https://mail.163.com/dashi/dlpro.html?from=mail81> ??
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/11e90c62/attachment.html>

From diogo.ferreira-sabino at univ-tlse3.fr  Tue Apr  9 09:36:37 2019
From: diogo.ferreira-sabino at univ-tlse3.fr (Diogo FERREIRA SABINO)
Date: Tue, 9 Apr 2019 16:36:37 +0200 (CEST)
Subject: [petsc-users] Constructing a MATNEST with blocks defined in
 different procs
In-Reply-To: <CADOhEh5whnESzNaf3JmP-j8DfzVVFDBary=6Ur2e6VtNSA-tiQ@mail.gmail.com>
References: <429247112.120532.1554463174712@scout-f5.univ-toulouse.fr>
	<CADOhEh5whnESzNaf3JmP-j8DfzVVFDBary=6Ur2e6VtNSA-tiQ@mail.gmail.com>
Message-ID: <1524658059.238250.1554820597850@scout-f5.univ-toulouse.fr>

Hi Mark,
Thank you for the quick answer.

So, I defined each block of the Nest matrix as a MATMPIAIJ in the World communicator and I set the local sizes in such a way that A00 and A11 are in the processors 0 and 1, respectively.
However, now I'm not able to set the off diagonal blocks of the Nest matrix. For example, if I try to set the block (0,1) of the nest with A00, the following error occurs:

[0]PETSC ERROR: Arguments are incompatible
[0]PETSC ERROR: Local sizes (2,2) of nested submatrix (0,1) do not agree with space defined by index sets (2,0)

I'm using the following lines to generate the Nest matrix:

? AfullNESTpointer[0]=Ablocks[0];
? AfullNESTpointer[1]=Ablocks[0];?  //Not working
? // AfullNESTpointer[1]=NULL;?????? //It works
? AfullNESTpointer[2]=NULL;
? AfullNESTpointer[3]=Ablocks[1];

? MatCreate(PETSC_COMM_WORLD,&AfullNEST);
? MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
? MatCreateNest(PETSC_COMM_WORLD,2,NULL,2,NULL,AfullNESTpointer,&AfullNEST);

A simple test is given below, lunching it with: mpirun -n 2 ./Main_petsc.exe

Thanks in advance,
Diogo

static char help[]???? = "Create MPI Nest";
#include <petscmat.h>
#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
? PetscInitialize(&argc,&argv,(char*)0,help);
? //////////////////////////////////////////////////////////////////////////////
? PetscErrorCode? ierr;
? PetscInt??????? i,PetscIntTEMP;
? //////////////////////////////////////////////////////////////////////////////
? PetscMPIInt MPIrank,MPIsize;
? MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
? MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);
? ///////////////////////////////////////////////////////////////?? Create Mats:
? Mat Ablocks[MPIsize];

? for(i=0;i<MPIsize;i++){
??? MatCreate(PETSC_COMM_WORLD,&Ablocks[i]);

??? PetscIntTEMP = (i==MPIrank) ? 2 : 0;
??? MatSetSizes(Ablocks[i],PetscIntTEMP,PetscIntTEMP,PETSC_DECIDE,PETSC_DECIDE);
??? MatSetType(Ablocks[i],MATMPIAIJ);
??? MatMPIAIJSetPreallocation(Ablocks[i],PetscIntTEMP,NULL,PetscIntTEMP,NULL);

??? PetscInt ISstart,ISend;
??? MatGetOwnershipRange(Ablocks[i],&ISstart,&ISend); //Check info
??? PetscSynchronizedPrintf(PETSC_COMM_WORLD,"proc(%d): Ablocks[%d]:(%d to %d) \n",MPIrank,i,ISstart,ISend);
? }
? PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT);

? MatSetValue(Ablocks[MPIrank],0,0,5+MPIrank,INSERT_VALUES);
? MatSetValue(Ablocks[MPIrank],0,1,10+MPIrank,INSERT_VALUES);
? MatSetValue(Ablocks[MPIrank],1,0,15+MPIrank,INSERT_VALUES);
? MatSetValue(Ablocks[MPIrank],1,1,20+MPIrank,INSERT_VALUES);

? for(i=0;i<MPIsize;i++){
??? MatAssemblyBegin(Ablocks[i],MAT_FINAL_ASSEMBLY);???? MatAssemblyEnd(Ablocks[i],MAT_FINAL_ASSEMBLY);
??? // MatView(Ablocks[i],PETSC_VIEWER_STDOUT_WORLD);
? }
? ///////////////////////////////////////////////////////////////?? Create Nest:
? Mat?????? AfullNEST, *AfullNESTpointer;
? Mat?????? AfullConverted;

? PetscMalloc1(4,&AfullNESTpointer);
? AfullNESTpointer[0]=Ablocks[0];
? AfullNESTpointer[1]=Ablocks[0];?  //Not working
? // AfullNESTpointer[1]=NULL;?????? //It works
? AfullNESTpointer[2]=NULL;
? AfullNESTpointer[3]=Ablocks[1];

? MatCreate(PETSC_COMM_WORLD,&AfullNEST);
? // MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
? MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
? MatCreateNest(PETSC_COMM_WORLD,2,NULL,2,NULL,AfullNESTpointer,&AfullNEST);

? MatView(AfullNEST,PETSC_VIEWER_STDOUT_WORLD);
? MatConvert(AfullNEST,MATMPIAIJ,MAT_INITIAL_MATRIX,&AfullConverted);
? MatView(AfullConverted,PETSC_VIEWER_STDOUT_WORLD);

? ierr = PetscFinalize(); CHKERRQ(ierr);? return 0;
}

April 5, 2019 2:38:46 PM CEST Mark Adams <mfadams at lbl.gov> wrote:

On Fri, Apr 5, 2019 at 7:19 AM Diogo FERREIRA SABINO via petsc-users <petsc-users at mcs.anl.gov> wrote:

Hi,
I'm new in petsc and I'm trying to construct a MATNEST in two procs, by setting each block of the nested matrix with a MATMPIAIJ matrix defined in each proc.
I'm trying to use MatCreateNest or MatNestSetSubMats, but I'm not being able to do it.
Using MatNestSetSubMats, I'm trying to construct the MATNEST, giving a pointer to the correct matrices depending on the MPIrank of that proc.
I'm obtaining the error message for the line :MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Wrong type of object: Parameter # 5

Is there a way of doing it, or all the blocks of the MATNEST have to exist in the same communicator as the MATNEST matrix?

Yes, they must all have the same communicator. A matrix can be empty on a process, so you just create them with the global communicator, set the local sizes that you want (eg, 0 on some procs).
?
A simple test is given below, lunching it with: mpirun -n 2 ./Main_petsc.exe

static char help[]? ? ?= "Create MPI Nest";
#include <petscmat.h>

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
? PetscInitialize(&argc,&argv,(char*)0,help);
? //////////////////////////////////////////////////////////////////////////////
? PetscErrorCode? ierr;
? PetscMPIInt MPIrank,MPIsize;
? MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
? MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);

? ////////////////////////////////////////////////////////? ?Create Each Matrix:
? Mat Adiag;

? //Create a Adiag different on each proc:
? ierr = MatCreate(PETSC_COMM_SELF,&Adiag);? ? ? ? ? ? ? ? ? ? ?CHKERRQ(ierr);
? ierr = MatSetSizes(Adiag,2,2,PETSC_DECIDE,PETSC_DECIDE);? ? ? CHKERRQ(ierr);
? ierr = MatSetType(Adiag,MATMPIAIJ);? ? ? ? ? ? ? ? ? ? ? ? ? ?CHKERRQ(ierr);
? ierr = MatSetFromOptions(Adiag);? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CHKERRQ(ierr);
? ierr = MatMPIAIJSetPreallocation(Adiag,2,NULL,2,NULL);? ? ? ? CHKERRQ(ierr);

? MatSetValue(Adiag,0,0,(MPIrank+5),INSERT_VALUES);
? MatSetValue(Adiag,0,1,(MPIrank+10),INSERT_VALUES);
? MatSetValue(Adiag,1,0,(MPIrank+15),INSERT_VALUES);
? MatSetValue(Adiag,1,1,(MPIrank+20),INSERT_VALUES);
? MatAssemblyBegin(Adiag,MAT_FINAL_ASSEMBLY);? ? ?MatAssemblyEnd(Adiag,MAT_FINAL_ASSEMBLY);

? ///////////////////////////////////////////////////////////////? ?Create Nest:
? MPI_Barrier(PETSC_COMM_WORLD);
? Mat? ? ? ?AfullNEST, *AfullNESTpointer;

? PetscMalloc1(2,&AfullNESTpointer);
? AfullNESTpointer[0]=NULL;
? AfullNESTpointer[1]=NULL;
? AfullNESTpointer[MPIrank]=Adiag;
? // Rank=0 --> AfullNESTpointer[0]=Adiag; AfullNESTpointer[1]=NULL;
? // Rank=1 --> AfullNESTpointer[0]=NULL;? AfullNESTpointer[1]=Adiag;

? IS? ? ? ? IS_ROW,IS_COL;
? ISCreateStride(PETSC_COMM_SELF,1,MPIrank,0,&IS_ROW);
? ISCreateStride(PETSC_COMM_SELF,2,0,1,&IS_COL);
? // Rank=0 --> IS_ROW= [ 0 ] ; IS_COL= [ 0, 1 ] ;
? // Rank=1 --> IS_ROW= [ 1 ] ; IS_COL= [ 0, 1 ] ;

? MatCreate(PETSC_COMM_WORLD,&AfullNEST);
? MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
? // MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
? // MatCreateNest(PETSC_COMM_WORLD,1,&IS_ROW,1,&IS_COL,AfullNESTpointer,&AfullNEST);
? ? ierr = MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer); CHKERRQ(ierr);

? ierr = PetscFinalize(); CHKERRQ(ierr);
? return 0;
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/5223196c/attachment.html>

From hzhang at mcs.anl.gov  Tue Apr  9 10:23:20 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 9 Apr 2019 15:23:20 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<7b104336-a067-e679-23ec-2a89e0ba9bc4@idris.fr>
	<CADOhEh5MHFQOoHF24hOWhoAiLnqYuZO29hwKzHmMndicF1xj0g@mail.gmail.com>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
Message-ID: <CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>

Myriam,
Do you have 'execution time scalability' plot? Did you use '-mat_freeintermediatedatastructures' for PETSc 3.10.2?
We made several computing optimizations on MatPtAP(), which might trade memory for speed. It would be helpful to see a complete comparison.
Hong

On Tue, Apr 9, 2019 at 7:43 AM Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi,

in my first mail, I provided a memory scaling concerning the PETSc
example #42. You'll find attached the main files used (one for PETSc
3.6.4, one for PETSc 3.10.2), and the corresponding memory scaling.

In the main files, I modified the solver/preconditioner, so that it
corresponds to my problem. You'll find the modifications by searching
the keyword "TopBridge". In particular, I use GAMG.

Note that the example is about solving Stokes equation, so using GAMG
may not be adapted. However, the memory gap appears and that's the
point. No matter if the results are correct.

Are these scripts useful for you? Let me know.

Thanks,

Myriam


Le 04/04/19 ? 00:09, Jed Brown a ?crit :
> Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> writes:
>
>> Hi all,
>>
>> for your information, you'll find attached the comparison of the weak
>> memory scalings when using :
>>
>> - PETSc 3.6.4 (reference)
>> - PETSc 3.10.4 without specific options
>> - PETSc 3.10.4 with the three scalability options you mentionned
>>
>> Using the scalability options does improve the memory scaling. However,
>> the 3.6 version still has a better one...
> Yes, this still looks significant.  Is this an effect we can still
> reproduce with a PETSc example and/or using a memory profiler (such as
> massif or gperftools)?  I think it's important for us to narrow down
> what causes this difference (looks like almost 2x on your 1e8 problem
> size) so we can fix.

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/d01d8ba7/attachment-0001.html>

From mfadams at lbl.gov  Tue Apr  9 13:30:56 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 9 Apr 2019 14:30:56 -0400
Subject: [petsc-users] Constructing a MATNEST with blocks defined in
 different procs
In-Reply-To: <1524658059.238250.1554820597850@scout-f5.univ-toulouse.fr>
References: <429247112.120532.1554463174712@scout-f5.univ-toulouse.fr>
	<CADOhEh5whnESzNaf3JmP-j8DfzVVFDBary=6Ur2e6VtNSA-tiQ@mail.gmail.com>
	<1524658059.238250.1554820597850@scout-f5.univ-toulouse.fr>
Message-ID: <CADOhEh6Ad_Cv75Jq6aQt72uZMNpUvKNRSjVr9cTLWqTGAiE1yA@mail.gmail.com>

On Tue, Apr 9, 2019 at 10:36 AM Diogo FERREIRA SABINO <
diogo.ferreira-sabino at univ-tlse3.fr> wrote:

> Hi Mark,
> Thank you for the quick answer.
>
> So, I defined each block of the Nest matrix as a MATMPIAIJ in the World
> communicator and I set the local sizes in such a way that A00 and A11 are
> in the processors 0 and 1, respectively.
> However, now I'm not able to set the off diagonal blocks of the Nest
> matrix. For example, if I try to set the block (0,1) of the nest with A00,
> the following error occurs:
>
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: Local sizes (2,2) of nested submatrix (0,1) do not agree
> with space defined by index sets (2,0)
>
> I'm using the following lines to generate the Nest matrix:
>
>   AfullNESTpointer[0]=Ablocks[0];
>   AfullNESTpointer[1]=Ablocks[0];  //Not working
>   // AfullNESTpointer[1]=NULL;       //It works
>   AfullNESTpointer[2]=NULL;
>   AfullNESTpointer[3]=Ablocks[1];
>
>   MatCreate(PETSC_COMM_WORLD,&AfullNEST);
>   MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
>
> MatCreateNest(PETSC_COMM_WORLD,2,NULL,2,NULL,AfullNESTpointer,&AfullNEST);
>
>
This is not how you create a MatNest. See:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html

The C example look pretty much like what you want to do:

MatCreateNest <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html#MatCreateNest>(PETSC_COMM_WORLD
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_COMM_WORLD.html#PETSC_COMM_WORLD>,
2, NULL, 2, NULL, s->subA, &s->A);




> A simple test is given below, lunching it with: mpirun -n 2
> ./Main_petsc.exe
>
> Thanks in advance,
> Diogo
>
>
>
> static char help[]     = "Create MPI Nest";
> #include <petscmat.h>
> #undef __FUNCT__
> #define __FUNCT__ "main"
> int main(int argc,char **argv)
> {
>   PetscInitialize(&argc,&argv,(char*)0,help);
>
> //////////////////////////////////////////////////////////////////////////////
>   PetscErrorCode  ierr;
>   PetscInt        i,PetscIntTEMP;
>
> //////////////////////////////////////////////////////////////////////////////
>   PetscMPIInt MPIrank,MPIsize;
>   MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
>   MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);
>   ///////////////////////////////////////////////////////////////   Create
> Mats:
>   Mat Ablocks[MPIsize];
>
>   for(i=0;i<MPIsize;i++){
>     MatCreate(PETSC_COMM_WORLD,&Ablocks[i]);
>
>     PetscIntTEMP = (i==MPIrank) ? 2 : 0;
>
> MatSetSizes(Ablocks[i],PetscIntTEMP,PetscIntTEMP,PETSC_DECIDE,PETSC_DECIDE);
>     MatSetType(Ablocks[i],MATMPIAIJ);
>
> MatMPIAIJSetPreallocation(Ablocks[i],PetscIntTEMP,NULL,PetscIntTEMP,NULL);
>
>     PetscInt ISstart,ISend;
>     MatGetOwnershipRange(Ablocks[i],&ISstart,&ISend); //Check info
>     PetscSynchronizedPrintf(PETSC_COMM_WORLD,"proc(%d): Ablocks[%d]:(%d to
> %d) \n",MPIrank,i,ISstart,ISend);
>   }
>   PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT);
>
>   MatSetValue(Ablocks[MPIrank],0,0,5+MPIrank,INSERT_VALUES);
>   MatSetValue(Ablocks[MPIrank],0,1,10+MPIrank,INSERT_VALUES);
>   MatSetValue(Ablocks[MPIrank],1,0,15+MPIrank,INSERT_VALUES);
>   MatSetValue(Ablocks[MPIrank],1,1,20+MPIrank,INSERT_VALUES);
>
>   for(i=0;i<MPIsize;i++){
>     MatAssemblyBegin(Ablocks[i],MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(Ablocks[i],MAT_FINAL_ASSEMBLY);
>     // MatView(Ablocks[i],PETSC_VIEWER_STDOUT_WORLD);
>   }
>   ///////////////////////////////////////////////////////////////   Create
> Nest:
>   Mat       AfullNEST, *AfullNESTpointer;
>   Mat       AfullConverted;
>
>   PetscMalloc1(4,&AfullNESTpointer);
>   AfullNESTpointer[0]=Ablocks[0];
>   AfullNESTpointer[1]=Ablocks[0];  //Not working
>   // AfullNESTpointer[1]=NULL;       //It works
>   AfullNESTpointer[2]=NULL;
>   AfullNESTpointer[3]=Ablocks[1];
>
>   MatCreate(PETSC_COMM_WORLD,&AfullNEST);
>   // MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
>   MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
>
> MatCreateNest(PETSC_COMM_WORLD,2,NULL,2,NULL,AfullNESTpointer,&AfullNEST);
>
>   MatView(AfullNEST,PETSC_VIEWER_STDOUT_WORLD);
>   MatConvert(AfullNEST,MATMPIAIJ,MAT_INITIAL_MATRIX,&AfullConverted);
>   MatView(AfullConverted,PETSC_VIEWER_STDOUT_WORLD);
>
>   ierr = PetscFinalize(); CHKERRQ(ierr);  return 0;
> }
> April 5, 2019 2:38:46 PM CEST Mark Adams <mfadams at lbl.gov> wrote:
>
>
>
> On Fri, Apr 5, 2019 at 7:19 AM Diogo FERREIRA SABINO via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi,
> I'm new in petsc and I'm trying to construct a MATNEST in two procs, by
> setting each block of the nested matrix with a MATMPIAIJ matrix defined in
> each proc.
> I'm trying to use MatCreateNest or MatNestSetSubMats, but I'm not being
> able to do it.
> Using MatNestSetSubMats, I'm trying to construct the MATNEST, giving a
> pointer to the correct matrices depending on the MPIrank of that proc.
> I'm obtaining the error message for the line
> :MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Wrong type of object: Parameter # 5
>
> Is there a way of doing it, or all the blocks of the MATNEST have to exist
> in the same communicator as the MATNEST matrix?
>
>
> Yes, they must all have the same communicator. A matrix can be empty on a
> process, so you just create them with the global communicator, set the
> local sizes that you want (eg, 0 on some procs).
>
>
> A simple test is given below, lunching it with: mpirun -n 2
> ./Main_petsc.exe
>
> static char help[]     = "Create MPI Nest";
> #include <petscmat.h>
>
> #undef __FUNCT__
> #define __FUNCT__ "main"
> int main(int argc,char **argv)
> {
>   PetscInitialize(&argc,&argv,(char*)0,help);
>
> //////////////////////////////////////////////////////////////////////////////
>   PetscErrorCode  ierr;
>   PetscMPIInt MPIrank,MPIsize;
>   MPI_Comm_rank(PETSC_COMM_WORLD,&MPIrank);
>   MPI_Comm_size(PETSC_COMM_WORLD,&MPIsize);
>
>   ////////////////////////////////////////////////////////   Create Each
> Matrix:
>   Mat Adiag;
>
>   //Create a Adiag different on each proc:
>   ierr = MatCreate(PETSC_COMM_SELF,&Adiag);
>  CHKERRQ(ierr);
>   ierr = MatSetSizes(Adiag,2,2,PETSC_DECIDE,PETSC_DECIDE);
> CHKERRQ(ierr);
>   ierr = MatSetType(Adiag,MATMPIAIJ);
>  CHKERRQ(ierr);
>   ierr = MatSetFromOptions(Adiag);
> CHKERRQ(ierr);
>   ierr = MatMPIAIJSetPreallocation(Adiag,2,NULL,2,NULL);
> CHKERRQ(ierr);
>
>   MatSetValue(Adiag,0,0,(MPIrank+5),INSERT_VALUES);
>   MatSetValue(Adiag,0,1,(MPIrank+10),INSERT_VALUES);
>   MatSetValue(Adiag,1,0,(MPIrank+15),INSERT_VALUES);
>   MatSetValue(Adiag,1,1,(MPIrank+20),INSERT_VALUES);
>   MatAssemblyBegin(Adiag,MAT_FINAL_ASSEMBLY);
>  MatAssemblyEnd(Adiag,MAT_FINAL_ASSEMBLY);
>
>   ///////////////////////////////////////////////////////////////   Create
> Nest:
>   MPI_Barrier(PETSC_COMM_WORLD);
>   Mat       AfullNEST, *AfullNESTpointer;
>
>   PetscMalloc1(2,&AfullNESTpointer);
>   AfullNESTpointer[0]=NULL;
>   AfullNESTpointer[1]=NULL;
>   AfullNESTpointer[MPIrank]=Adiag;
>   // Rank=0 --> AfullNESTpointer[0]=Adiag; AfullNESTpointer[1]=NULL;
>   // Rank=1 --> AfullNESTpointer[0]=NULL;  AfullNESTpointer[1]=Adiag;
>
>   IS        IS_ROW,IS_COL;
>   ISCreateStride(PETSC_COMM_SELF,1,MPIrank,0,&IS_ROW);
>   ISCreateStride(PETSC_COMM_SELF,2,0,1,&IS_COL);
>   // Rank=0 --> IS_ROW= [ 0 ] ; IS_COL= [ 0, 1 ] ;
>   // Rank=1 --> IS_ROW= [ 1 ] ; IS_COL= [ 0, 1 ] ;
>
>   MatCreate(PETSC_COMM_WORLD,&AfullNEST);
>   MatSetSizes(AfullNEST,2,2,PETSC_DECIDE,PETSC_DECIDE);
>   // MatSetSizes(AfullNEST,PETSC_DECIDE,PETSC_DECIDE,4,4);
>   //
> MatCreateNest(PETSC_COMM_WORLD,1,&IS_ROW,1,&IS_COL,AfullNESTpointer,&AfullNEST);
>     ierr =
> MatNestSetSubMats(AfullNEST,1,&IS_ROW,2,&IS_COL,AfullNESTpointer);
> CHKERRQ(ierr);
>
>   ierr = PetscFinalize(); CHKERRQ(ierr);
>   return 0;
> }
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190409/0f81e107/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr  9 16:19:03 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 9 Apr 2019 21:19:03 +0000
Subject: [petsc-users] How to decompose and record a constant A
In-Reply-To: <CADOhEh5vkK9Hw4rR+t2EPD2wHmy-XrYUKbyJRD75JN=TaF8arQ@mail.gmail.com>
References: <ABA46849-BEA2-456F-BB3B-368B33D8BDB8@gmail.com>
	<CADOhEh5vkK9Hw4rR+t2EPD2wHmy-XrYUKbyJRD75JN=TaF8arQ@mail.gmail.com>
Message-ID: <ED0DB7C5-6AFB-444D-BE7A-4DF793077A46@anl.gov>



> On Apr 9, 2019, at 3:33 AM, Mark Adams via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> PETSc solvers will cache any setup computations, like a matrix factorization, and will reuse them on subsequent solves. 
> 
> If you call KSPSetOperators, PETSc will assume its cache is invalid and redo any required setup on subsequent solves.

    This is not actually correct; calling KSPSetOperators() again with the same matrix where you have not changed the matrix values will
not require any redo of the preconditioner setup.  KSP/PC/Mat tracks the current state of the matrix and rebuilds the preconditioner ONLY 
if you have changed the values of the matrix somehow (for example with new calls to MatSetValues, MatDiagonalSet etc). 

    Thus you can call KSPSetOperators() once with your matrix and make repeated calls to KSPSolve(), if the matrix is unchanged KSPSolve() will reuse the previously constructed preconditioner, if the matrix is changed PETSc will automatically redo the construction of the preconditioner.

   Barry

> 
> Mark
> 
> On Tue, Apr 9, 2019 at 4:08 AM ztdepyahoo via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Dear group:
>       I need to solve a Laplacian problem Ax=b every time step where A is a constant . I want to store the inverse of A at the first time step. 
> how to do with Petsc
>  Regards
> 
>  	
> ztdepyahoo
> ztdepyahoo at gmail.com
> ??? ?????? ??


From zulfi6000 at yahoo.com  Tue Apr  9 23:31:17 2019
From: zulfi6000 at yahoo.com (Zulfi Khan)
Date: Wed, 10 Apr 2019 04:31:17 +0000 (UTC)
Subject: [petsc-users] Required Help on Calculation of All-to-All broadcast
 in a Balanced Binary Tree -Please
References: <89337697.919951.1554870677870.ref@mail.yahoo.com>
Message-ID: <89337697.919951.1554870677870@mail.yahoo.com>


Calculation of Cost of all-to-broadcast for a Balanced Binary Tree

Hi,

I have a question is:

Given a balanced binary tree as shown in Figure 4.7 (attached),describe a procedure to perform all-to-all broadcast that takes time (ts +twmp/2)logp for m-word messages on p nodes (assuming th is ignored). Assumethat only the leaves of the tree contain nodes, and that an exchange of twom-word messages between any two nodes connected by bidirectional channels takestime ts + twmk if the communication channel (or a part of it) is shared by ksimultaneous messages.

?

Equations:

comm cost = ts + tw* m* k (as given in the question)

I am trying to calculate the cost of balanced binary tree for allto all broadcast. I have created a procedure for this broadcast:

?P0 P1 P2 P3 ?(left sub tree): P4, P5, P6, P7(right sub tree)

1st Step: broadcast from left subtree to right subtree(for 8 processors four on left subtree and four on right subtree), the cost is:




P0 exchanges with P7 (i.e each P0 & P7 contains 2 messages)

P1 exchanges with P6 (i.e each P1 & P6 contains 2 messages) 

P2 exchanges with P5 (i.e. each P2 & P5 contains 2 messages)

P3 exchanges with P4 (i.e each P3 and P4 contains 2 messages)

2nd Step: Broadcast Across? the Subtrees Within the Left and RightSubtree

P0 exchanges with P3 (i.e each P0 & P3 contains 4 messages)

P1 exchanges with P2 (i.e each P1 & P2 contains 4 messages)

P4 exchanges with P7 (i.e. each P4 & P7 contains 4 messages)

P5 exchanges with P6 (i.e. each P5 & P6 contains 4 messages) ?

3rd Step: Broadcast Within?the Subtrees of Left and Right Subtree

P0 exchanges with P1 (i.e each P0 & P1 contains 8 messages)

P2 exchanges with P3 (i.e each P2 & P3 contains 8 messages)

P4 exchanges with P5 ((i.e each P4 & P5 contains 8 messages)

P6 exchanges with P7 (i.e. each P6 & P7 contains 8 messages)

Step 1: t_s + t_w m * 2 //Let k =2.Is k correct?

Step 2: t_s + t_w m *? 1? //Letk = 1. Is? k correct?

Step3: t_s +t_w m * 1// Let k = 1. Is k ?correct?

logp(t_s + 2 * t_w m + 1 * t_w m +1 * t_w m)

?t_comm = logp(t_s+4t_w m)

Now p/2 = 4

There t_comm = logp(t_s + t_w mp/2)

?

My answer is correct as given inthe question but I don?t have any justification for choosing the value of ?k??

Can some body please guide me is mysolution correct? What is the justification of choosing the value of k?

?

Stack Exchange link is;
https://cs.stackexchange.com/questions/106631/all-to-all-broadcast-on-a-balanced-binary-tree

Kindly help me, I would be very much thankful to you guys.

Zulfi.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/45e6a2c6/attachment-0001.html>

From jed at jedbrown.org  Tue Apr  9 23:52:46 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 09 Apr 2019 22:52:46 -0600
Subject: [petsc-users] Required Help on Calculation of All-to-All
 broadcast in a Balanced Binary Tree -Please
In-Reply-To: <89337697.919951.1554870677870@mail.yahoo.com>
References: <89337697.919951.1554870677870.ref@mail.yahoo.com>
	<89337697.919951.1554870677870@mail.yahoo.com>
Message-ID: <87wok28ns1.fsf@jedbrown.org>

This sounds an awful lot like a homework question.  In any case, it does
not relate directly to PETSc and is thus off-topic for this list.

Zulfi Khan via petsc-users <petsc-users at mcs.anl.gov> writes:

> Calculation of Cost of all-to-broadcast for a Balanced Binary Tree
>
> Hi,
>
> I have a question is:
>
> Given a balanced binary tree as shown in Figure 4.7 (attached),describe a procedure to perform all-to-all broadcast that takes time (ts +twmp/2)logp for m-word messages on p nodes (assuming th is ignored). Assumethat only the leaves of the tree contain nodes, and that an exchange of twom-word messages between any two nodes connected by bidirectional channels takestime ts + twmk if the communication channel (or a part of it) is shared by ksimultaneous messages.
>
> ?
>
> Equations:
>
> comm cost = ts + tw* m* k (as given in the question)
>
> I am trying to calculate the cost of balanced binary tree for allto all broadcast. I have created a procedure for this broadcast:
>
> ?P0 P1 P2 P3 ?(left sub tree): P4, P5, P6, P7(right sub tree)
>
> 1st Step: broadcast from left subtree to right subtree(for 8 processors four on left subtree and four on right subtree), the cost is:
>
>
>
>
> P0 exchanges with P7 (i.e each P0 & P7 contains 2 messages)
>
> P1 exchanges with P6 (i.e each P1 & P6 contains 2 messages) 
>
> P2 exchanges with P5 (i.e. each P2 & P5 contains 2 messages)
>
> P3 exchanges with P4 (i.e each P3 and P4 contains 2 messages)
>
> 2nd Step: Broadcast Across? the Subtrees Within the Left and RightSubtree
>
> P0 exchanges with P3 (i.e each P0 & P3 contains 4 messages)
>
> P1 exchanges with P2 (i.e each P1 & P2 contains 4 messages)
>
> P4 exchanges with P7 (i.e. each P4 & P7 contains 4 messages)
>
> P5 exchanges with P6 (i.e. each P5 & P6 contains 4 messages) ?
>
> 3rd Step: Broadcast Within?the Subtrees of Left and Right Subtree
>
> P0 exchanges with P1 (i.e each P0 & P1 contains 8 messages)
>
> P2 exchanges with P3 (i.e each P2 & P3 contains 8 messages)
>
> P4 exchanges with P5 ((i.e each P4 & P5 contains 8 messages)
>
> P6 exchanges with P7 (i.e. each P6 & P7 contains 8 messages)
>
> Step 1: t_s + t_w m * 2 //Let k =2.Is k correct?
>
> Step 2: t_s + t_w m *? 1? //Letk = 1. Is? k correct?
>
> Step3: t_s +t_w m * 1// Let k = 1. Is k ?correct?
>
> logp(t_s + 2 * t_w m + 1 * t_w m +1 * t_w m)
>
> ?t_comm = logp(t_s+4t_w m)
>
> Now p/2 = 4
>
> There t_comm = logp(t_s + t_w mp/2)
>
> ?
>
> My answer is correct as given inthe question but I don?t have any justification for choosing the value of ?k??
>
> Can some body please guide me is mysolution correct? What is the justification of choosing the value of k?
>
> ?
>
> Stack Exchange link is;
> https://cs.stackexchange.com/questions/106631/all-to-all-broadcast-on-a-balanced-binary-tree
>
> Kindly help me, I would be very much thankful to you guys.
>
> Zulfi.

From myriam.peyrounette at idris.fr  Wed Apr 10 07:12:23 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Wed, 10 Apr 2019 14:12:23 +0200
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
Message-ID: <3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>

Here is the time weak scaling from the same study. The 3.10.2 version
seems to be much more stable with regard to the execution time. But not
necessarily faster for "large scale" simulations (problem size = 1e8).

I didn't use -mat_freeintermediatedatastructures. I tested it this
morning and the solver diverges when using this option (KSPReason -3).

Myriam


Le 04/09/19 ? 17:23, Zhang, Hong a ?crit?:
> Myriam,
> Do you have 'execution time scalability' plot? Did you use
> '-mat_freeintermediatedatastructures' for PETSc 3.10.2?
> We made several computing optimizations on MatPtAP(), which might
> trade memory for speed. It would be helpful to see a complete comparison.
> Hong
>
> On Tue, Apr 9, 2019 at 7:43 AM Myriam Peyrounette via petsc-users
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Hi,
>
>     in my first mail, I provided a memory scaling concerning the PETSc
>     example #42. You'll find attached the main files used (one for PETSc
>     3.6.4, one for PETSc 3.10.2), and the corresponding memory scaling.
>
>     In the main files, I modified the solver/preconditioner, so that it
>     corresponds to my problem. You'll find the modifications by searching
>     the keyword "TopBridge". In particular, I use GAMG.
>
>     Note that the example is about solving Stokes equation, so using GAMG
>     may not be adapted. However, the memory gap appears and that's the
>     point. No matter if the results are correct.
>
>     Are these scripts useful for you? Let me know.
>
>     Thanks,
>
>     Myriam
>
>
>     Le 04/04/19 ? 00:09, Jed Brown a ?crit?:
>     > Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov
>     <mailto:petsc-users at mcs.anl.gov>> writes:
>     >
>     >> Hi all,
>     >>
>     >> for your information, you'll find attached the comparison of
>     the weak
>     >> memory scalings when using :
>     >>
>     >> - PETSc 3.6.4 (reference)
>     >> - PETSc 3.10.4 without specific options
>     >> - PETSc 3.10.4 with the three scalability options you mentionned
>     >>
>     >> Using the scalability options does improve the memory scaling.
>     However,
>     >> the 3.6 version still has a better one...
>     > Yes, this still looks significant.? Is this an effect we can still
>     > reproduce with a PETSc example and/or using a memory profiler
>     (such as
>     > massif or gperftools)?? I think it's important for us to narrow down
>     > what causes this difference (looks like almost 2x on your 1e8
>     problem
>     > size) so we can fix.
>
>     -- 
>     Myriam Peyrounette
>     CNRS/IDRIS - HLST
>     --
>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/928342e0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: time_scaling_big_cases_ex42_irene.png
Type: image/png
Size: 16858 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/928342e0/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/928342e0/attachment-0001.p7s>

From mfadams at lbl.gov  Wed Apr 10 07:21:41 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 10 Apr 2019 08:21:41 -0400
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
Message-ID: <CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>

This looks like it might be noisy data. I'd make sure you run each size on
the same set of nodes and you might run each job twice (A,B,A,B) in a job
script.

On Wed, Apr 10, 2019 at 8:12 AM Myriam Peyrounette via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Here is the time weak scaling from the same study. The 3.10.2 version
> seems to be much more stable with regard to the execution time. But not
> necessarily faster for "large scale" simulations (problem size = 1e8).
>
> I didn't use -mat_freeintermediatedatastructures. I tested it this morning
> and the solver diverges when using this option (KSPReason -3).
>
> Myriam
>
> Le 04/09/19 ? 17:23, Zhang, Hong a ?crit :
>
> Myriam,
> Do you have 'execution time scalability' plot? Did you use
> '-mat_freeintermediatedatastructures' for PETSc 3.10.2?
> We made several computing optimizations on MatPtAP(), which might trade
> memory for speed. It would be helpful to see a complete comparison.
> Hong
>
> On Tue, Apr 9, 2019 at 7:43 AM Myriam Peyrounette via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> in my first mail, I provided a memory scaling concerning the PETSc
>> example #42. You'll find attached the main files used (one for PETSc
>> 3.6.4, one for PETSc 3.10.2), and the corresponding memory scaling.
>>
>> In the main files, I modified the solver/preconditioner, so that it
>> corresponds to my problem. You'll find the modifications by searching
>> the keyword "TopBridge". In particular, I use GAMG.
>>
>> Note that the example is about solving Stokes equation, so using GAMG
>> may not be adapted. However, the memory gap appears and that's the
>> point. No matter if the results are correct.
>>
>> Are these scripts useful for you? Let me know.
>>
>> Thanks,
>>
>> Myriam
>>
>>
>> Le 04/04/19 ? 00:09, Jed Brown a ?crit :
>> > Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov> writes:
>> >
>> >> Hi all,
>> >>
>> >> for your information, you'll find attached the comparison of the weak
>> >> memory scalings when using :
>> >>
>> >> - PETSc 3.6.4 (reference)
>> >> - PETSc 3.10.4 without specific options
>> >> - PETSc 3.10.4 with the three scalability options you mentionned
>> >>
>> >> Using the scalability options does improve the memory scaling. However,
>> >> the 3.6 version still has a better one...
>> > Yes, this still looks significant.  Is this an effect we can still
>> > reproduce with a PETSc example and/or using a memory profiler (such as
>> > massif or gperftools)?  I think it's important for us to narrow down
>> > what causes this difference (looks like almost 2x on your 1e8 problem
>> > size) so we can fix.
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/fcca8b51/attachment.html>

From vu.doquochust at gmail.com  Wed Apr 10 08:01:01 2019
From: vu.doquochust at gmail.com (Vu Do Quoc)
Date: Wed, 10 Apr 2019 20:01:01 +0700
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
Message-ID: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>

Hi all,

I am trying to insert Petsc to OpenFOAM opensource software.
I have been successfully compiling Petsc with an available solver in
OpenFOAM by linking it with the shared library libpetsc.so. However, when I
call the solver to run a test case, I got an error saying that:
"libpetsc.so cannot be found", even though the library still exists in the
$PETSC_ARCH/lib folder.

I have been struggling for weeks but still, have not been able to figure it
out. Therefore I would be very grateful for any suggestion to solve this
problem.

Thanks in advance for your time,

Best regards,

Vu Do
-- 

*Vu Q. Do*------------------------------------------------
*Student of Aeronautical Engineering*
Programme de Formation d'Ing?nieurs d'Excellence au Vietnam *- PFIEV*
School of Transportation Engineering
Hanoi University of Science and Technology
01 Dai Co Viet Avenue, Hanoi, Vietnam
E-mail: vu.doquochust at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/830cfc5c/attachment.html>

From balay at mcs.anl.gov  Wed Apr 10 08:23:43 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 10 Apr 2019 13:23:43 +0000
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
In-Reply-To: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
References: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904100822070.3207@sb>

Runtime error? You might have to add the path to $PETSC_ARCH/lib in  LD_LIBRARY_PATH env variable
or - to your link command. If linux/gcc - the linker option is -Wl,-rpath,$PETSC_ARCH/lib

If not - send detail logs.

Satish

On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:

> Hi all,
> 
> I am trying to insert Petsc to OpenFOAM opensource software.
> I have been successfully compiling Petsc with an available solver in
> OpenFOAM by linking it with the shared library libpetsc.so. However, when I
> call the solver to run a test case, I got an error saying that:
> "libpetsc.so cannot be found", even though the library still exists in the
> $PETSC_ARCH/lib folder.
> 
> I have been struggling for weeks but still, have not been able to figure it
> out. Therefore I would be very grateful for any suggestion to solve this
> problem.
> 
> Thanks in advance for your time,
> 
> Best regards,
> 
> Vu Do
> 


From m.colera at upm.es  Wed Apr 10 10:29:37 2019
From: m.colera at upm.es (Manuel Colera Rico)
Date: Wed, 10 Apr 2019 17:29:37 +0200
Subject: [petsc-users] Error with KSPSetUp and MatNest
Message-ID: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>

Hello,

I am trying to solve a system whose matrix is of type MatNest. If I 
don't use KSPSetUp(), everything is fine. However, if I use that 
routine, I get the following error:

0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of 
blocks.
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
[0]PETSC ERROR: 
/home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion 
on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
--with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native" 
CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 
-march=native -mtune=native" --with-debugging=0 --download-fblaslapack 
--download--f2cblaslapack --download-mpich --download--hypre 
--download-scalapack --download-mumps --download-suitesparse 
--download-ptscotch --download-pastix --with-matlab --with-openmp
[0]PETSC ERROR: #1 VecCopy_Nest() line 68 in 
/opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
[0]PETSC ERROR: #2 VecCopy() line 1614 in 
/opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #3 KSPInitialResidual() line 63 in 
/opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in 
/opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #5 KSPSolve() line 782 in 
/opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c

Please find attached a MWE (it is a slight modification of that of the 
post opened by Ce Qin, 
https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html, 
whose answer I have not found).

By the way, with the newest version of PETSc, Eclipse marks as errors 
the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it 
compiles and executes well. Perhaps it is a problem related to Eclipse, 
but this did not happen with the older versions of PETSc.

Thanks and regards,

Manuel

---

-------------- next part --------------
A non-text attachment was scrubbed...
Name: tests.c
Type: text/x-csrc
Size: 2751 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/94502cd5/attachment.bin>

From knepley at gmail.com  Wed Apr 10 10:41:12 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Apr 2019 11:41:12 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
Message-ID: <CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>

On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I am trying to solve a system whose matrix is of type MatNest. If I
> don't use KSPSetUp(), everything is fine. However, if I use that
> routine, I get the following error:
>
> 0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of
> blocks.
>

This seems self-explanatory. Nest vectors must have the same number of
blocks to be compatible.

More broadly, there should be no reason to use Nest vectors or matrices. It
is an optimization to
be used at the very end, only after you have profiled the code and seen
that its important. You can
do everything you want to do without ever touching Nest, and it looks like
the Nest interface is a
problem for your code right now.

  Thanks,

    Matt


> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
> [0]PETSC ERROR:
> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>
> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack
> --download--f2cblaslapack --download-mpich --download--hypre
> --download-scalapack --download-mumps --download-suitesparse
> --download-ptscotch --download-pastix --with-matlab --with-openmp
> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
> [0]PETSC ERROR: #2 VecCopy() line 1614 in
> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #5 KSPSolve() line 782 in
> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>
> Please find attached a MWE (it is a slight modification of that of the
> post opened by Ce Qin,
> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>
> whose answer I have not found).
>
> By the way, with the newest version of PETSc, Eclipse marks as errors
> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
> compiles and executes well. Perhaps it is a problem related to Eclipse,
> but this did not happen with the older versions of PETSc.
>
> Thanks and regards,
>
> Manuel
>
> ---
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/209cee38/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Apr 10 11:37:41 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 10 Apr 2019 16:37:41 +0000
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
In-Reply-To: <alpine.LFD.2.21.1904100822070.3207@sb>
References: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
	<alpine.LFD.2.21.1904100822070.3207@sb>
Message-ID: <56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov>


  We don't know much about OpenFoam but 

1)  if I do a 

  git grep -i petsc 

    in the https://develop.openfoam.com/Development/OpenFOAM-plus.git repository I see various configuration files specifically for PETSc. 

etc/config.csh/petsc  etc/config.sh/petsc wmake/scripts/have_petsc  

    so it appears that OpenFOAM has the tools to be linked against PETSc (to me the documentation on how to use them is rather terse). Are 
    you using these?  If you have trouble with them perhaps you can ask the OpenFOAM user community how to use them. 


  2)  if you are editing the Make/options file directly you can try changing 

     -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc

     to 

     -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib   -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc



   Note also that simply including petsc.h into the OpenFoam source code and linking against -lpetsc will not immediately allow calling the PETSc solvers from OpenFOAM. One needs to write all the interface code that sets up and calls the PETSc solvers from OpenFOAM. There is a paper https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework that describes at an abstract level how they wrote code that calls the PETSc solvers from OpenFOAM but the source code that actually does the work does not appear to be available.

   Note that PETSc is now at version 3.11 we recommend working with that version (unless you already have a lot of code that calls PETSc written with a previous version of PETSc, for that we recommend first upgrading to petsc 3.11 and then continuing to add code).

   Barry





> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Runtime error? You might have to add the path to $PETSC_ARCH/lib in  LD_LIBRARY_PATH env variable
> or - to your link command. If linux/gcc - the linker option is -Wl,-rpath,$PETSC_ARCH/lib
> 
> If not - send detail logs.
> 
> Satish
> 
> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
> 
>> Hi all,
>> 
>> I am trying to insert Petsc to OpenFOAM opensource software.
>> I have been successfully compiling Petsc with an available solver in
>> OpenFOAM by linking it with the shared library libpetsc.so. However, when I
>> call the solver to run a test case, I got an error saying that:
>> "libpetsc.so cannot be found", even though the library still exists in the
>> $PETSC_ARCH/lib folder.
>> 
>> I have been struggling for weeks but still, have not been able to figure it
>> out. Therefore I would be very grateful for any suggestion to solve this
>> problem.
>> 
>> Thanks in advance for your time,
>> 
>> Best regards,
>> 
>> Vu Do
>> 
> 


From m.colera at upm.es  Wed Apr 10 11:49:22 2019
From: m.colera at upm.es (Manuel Colera Rico)
Date: Wed, 10 Apr 2019 18:49:22 +0200
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
Message-ID: <da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>

Thank you for your answer, Matt. In the MWE example attached before, 
both Nest vectors (the r.h.s. of the system and the vector of unknowns) 
are composed of the same number of blocks (2). Indeed, PETSc is able to 
solve the system if KSPSetUp() is not called, so the 
system/MatNest/MatVec's must not incompatible at all. Therefore, I 
wonder if I have missed to called something before this routine or if 
this is a KSPSetUp's bug.

Of course one can always directly define a single matrix and a single 
vector, but I find it easier to work with Nest matrices and vectors. 
Moreover, I think that the moment to use them is from the beginning... 
once all the code is developed, it is very hard to switch matrices types.

Regards,

Manuel

---


On 4/10/19 5:41 PM, Matthew Knepley wrote:
> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Hello,
>
>     I am trying to solve a system whose matrix is of type MatNest. If I
>     don't use KSPSetUp(), everything is fine. However, if I use that
>     routine, I get the following error:
>
>     0]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [0]PETSC ERROR: Invalid argument
>     [0]PETSC ERROR: Nest vector arguments 1 and 2 have different
>     numbers of
>     blocks.
>
>
> This seems self-explanatory. Nest vectors must have the same number of 
> blocks to be compatible.
>
> More broadly, there should be no reason to use Nest vectors or 
> matrices. It is an optimization to
> be used at the very end, only after you have profiled the code and 
> seen that its important. You can
> do everything you want to do without ever touching Nest, and it looks 
> like the Nest interface is a
> problem for your code right now.
>
> ? Thanks,
>
> ? ? Matt
>
>     [0]PETSC ERROR: See
>     http://www.mcs.anl.gov/petsc/documentation/faq.html
>     for trouble shooting.
>     [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>     [0]PETSC ERROR:
>     /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>
>     on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
>     [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>     --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
>     CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>     -march=native -mtune=native" --with-debugging=0
>     --download-fblaslapack
>     --download--f2cblaslapack --download-mpich --download--hypre
>     --download-scalapack --download-mumps --download-suitesparse
>     --download-ptscotch --download-pastix --with-matlab --with-openmp
>     [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
>     /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>     [0]PETSC ERROR: #2 VecCopy() line 1614 in
>     /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>     [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
>     /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>     [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
>     /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>     [0]PETSC ERROR: #5 KSPSolve() line 782 in
>     /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>     [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>
>     Please find attached a MWE (it is a slight modification of that of
>     the
>     post opened by Ce Qin,
>     https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>
>     whose answer I have not found).
>
>     By the way, with the newest version of PETSc, Eclipse marks as errors
>     the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
>     compiles and executes well. Perhaps it is a problem related to
>     Eclipse,
>     but this did not happen with the older versions of PETSc.
>
>     Thanks and regards,
>
>     Manuel
>
>     ---
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/2391cd43/attachment.html>

From knepley at gmail.com  Wed Apr 10 11:57:03 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Apr 2019 12:57:03 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
Message-ID: <CAMYG4GnLVWkQB755o5EMTM4Zts-3MX1A-nEi-tQf-tOqdr--Lw@mail.gmail.com>

On Wed, Apr 10, 2019 at 12:49 PM Manuel Colera Rico <m.colera at upm.es> wrote:

> Thank you for your answer, Matt. In the MWE example attached before, both
> Nest vectors (the r.h.s. of the system and the vector of unknowns) are
> composed of the same number of blocks (2). Indeed, PETSc is able to solve
> the system if KSPSetUp() is not called, so the system/MatNest/MatVec's must
> not incompatible at all. Therefore, I wonder if I have missed to called
> something before this routine or if this is a KSPSetUp's bug.
>
I will run it.

> Of course one can always directly define a single matrix and a single
> vector, but I find it easier to work with Nest matrices and vectors.
> Moreover, I think that the moment to use them is from the beginning... once
> all the code is developed, it is very hard to switch matrices types.
>
This is not the case, and its the major reason why I want to remove
Vec/MatNest from the public interface. This is a major misconception. You
can
switch matrix types using -mat_type <type>. You can pull out submatrices
using MatGetSubmatrix() from MatAIJ just as simply as MatNest.
If you want to keep assembling submatrices, you can use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef

so that the assembly looks just like it does now for you, but it assembles
to an AIJ _or_ a Nest, depending on what you
choose at the command line. Then you can debug everything as AIJ, and
finally switch to Nest if you need it for optimization.

  Thanks,

     Matt

> Regards,
>
> Manuel
>
> ---
>
>
> On 4/10/19 5:41 PM, Matthew Knepley wrote:
>
> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello,
>>
>> I am trying to solve a system whose matrix is of type MatNest. If I
>> don't use KSPSetUp(), everything is fine. However, if I use that
>> routine, I get the following error:
>>
>> 0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Invalid argument
>> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of
>> blocks.
>>
>
> This seems self-explanatory. Nest vectors must have the same number of
> blocks to be compatible.
>
> More broadly, there should be no reason to use Nest vectors or matrices.
> It is an optimization to
> be used at the very end, only after you have profiled the code and seen
> that its important. You can
> do everything you want to do without ever touching Nest, and it looks like
> the Nest interface is a
> problem for your code right now.
>
>   Thanks,
>
>     Matt
>
>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>> [0]PETSC ERROR:
>> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>>
>> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
>> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack
>> --download--f2cblaslapack --download-mpich --download--hypre
>> --download-scalapack --download-mumps --download-suitesparse
>> --download-ptscotch --download-pastix --with-matlab --with-openmp
>> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>> [0]PETSC ERROR: #2 VecCopy() line 1614 in
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #5 KSPSolve() line 782 in
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>>
>> Please find attached a MWE (it is a slight modification of that of the
>> post opened by Ce Qin,
>> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>>
>> whose answer I have not found).
>>
>> By the way, with the newest version of PETSc, Eclipse marks as errors
>> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
>> compiles and executes well. Perhaps it is a problem related to Eclipse,
>> but this did not happen with the older versions of PETSc.
>>
>> Thanks and regards,
>>
>> Manuel
>>
>> ---
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/b1086f4e/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Wed Apr 10 12:58:39 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 10 Apr 2019 12:58:39 -0500
Subject: [petsc-users] Argument out of range error only in certain mpi sizes
Message-ID: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>

Hi PETSc developers,

I wanted to convert my code that in which I was using general Vec/Mat to
DMDA based grid management (nothing fancy just a 5-point complex stencil).
For this, I created a DA object and created the global solution vector
using this. This worked fine.

Now, I created the matrix using DMCreateMatrix and fill it using the
regular function. I get no error when I run the problem using mpirun -np 1
and I thought my matrix filling logic aligns with the DM non-zero locations
(Star Stencil, width 1). With mpirun -np 2, no errors either. But with
mpirun -np 4 or 8, I get errors which say : "Argument out of range/
Inserting a new nonzero at global row/column ... into matrix".

I would switch over the logic provided in KSP/ex46.c for filling the Matrix
via the MatSetStencil logic but I wanted to know what I was doing wrong in
my current code since I've never seen an error that depends on number of
mpi processes.

I'm attaching the code ( which works if the matrix is created without using
the DA, i.e. comment out line 159, uncomment 161/162 and I'm doing this on
a small grid to catch errors.).

Thanks in advance for the help.


-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/16651565/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_dmda.c
Type: application/octet-stream
Size: 15700 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/16651565/attachment.obj>

From knepley at gmail.com  Wed Apr 10 13:05:45 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Apr 2019 14:05:45 -0400
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
Message-ID: <CAMYG4G=OkYZDiQwMueJUjiYpfB0q5GihEWe7yhT8c8=Qcf86iA@mail.gmail.com>

On Wed, Apr 10, 2019 at 2:00 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi PETSc developers,
>
> I wanted to convert my code that in which I was using general Vec/Mat to
> DMDA based grid management (nothing fancy just a 5-point complex stencil).
> For this, I created a DA object and created the global solution vector
> using this. This worked fine.
>
> Now, I created the matrix using DMCreateMatrix and fill it using the
> regular function. I get no error when I run the problem using mpirun -np 1
> and I thought my matrix filling logic aligns with the DM non-zero locations
> (Star Stencil, width 1). With mpirun -np 2, no errors either. But with
> mpirun -np 4 or 8, I get errors which say : "Argument out of range/
> Inserting a new nonzero at global row/column ... into matrix".
>
> I would switch over the logic provided in KSP/ex46.c for filling the
> Matrix via the MatSetStencil logic but I wanted to know what I was doing
> wrong in my current code since I've never seen an error that depends on
> number of mpi processes.
>
> I'm attaching the code ( which works if the matrix is created without
> using the DA, i.e. comment out line 159, uncomment 161/162 and I'm doing
> this on a small grid to catch errors.).
>

You are assuming a lexicographic ordering of unknowns, but we do not do
that in parallel. There is a manual chapter
about the difference between PETSc and Natural numbering in DMDA.

  Thanks,

    Matt


> Thanks in advance for the help.
>
>
> --
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/b636a92d/attachment.html>

From bsmith at mcs.anl.gov  Wed Apr 10 13:22:21 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 10 Apr 2019 18:22:21 +0000
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
Message-ID: <477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>


   Sajid,

     This code won't work. You are assuming that neighbors to the north and south can be obtained from set_col = i + My and set_col = i - My. This is simply incorrect. The global number of unknowns in PETSc is by process (not natural ordering on the entire domain) this is why the PETSc examples that use DMDA us either MatSetValueStencil() or MatSetValuesLocal(). Since your problem has a simple stencil you should use the MatSetValuesStencil() approach and loop over the local part of the grid, not over the local rows of the matrix. Copy one of the src/ksp/ksp/examples/tutorials examples as a starting point.

   Barry

A side note not directly related to the current problem.  You are storing both physical quantities (lambs, energy) and grid quantities (mx, my ...) in the AppCtx. We don't recommend this, better to just store the physical quantities there and to retrieve the grid quantities in the code from the DMDA object whenever you need them (usually with calls to DMDAGetInfo() instead of hardwiring them into the AppCtx. The reason you want to separate the problem specific stuff from the grid specific stuff is so that your functions can be called for different DMDA (for examples calling RHSMatrixFreeSpace for compute the matrix for each level of multigrid) without having to change stuff in the AppCtx between each call to match the "current" grid.  Your current handling is not "wrong" it is just perhaps not the most desirable design.



> On Apr 10, 2019, at 12:58 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi PETSc developers, 
> 
> I wanted to convert my code that in which I was using general Vec/Mat to DMDA based grid management (nothing fancy just a 5-point complex stencil). For this, I created a DA object and created the global solution vector using this. This worked fine. 
> 
> Now, I created the matrix using DMCreateMatrix and fill it using the regular function. I get no error when I run the problem using mpirun -np 1 and I thought my matrix filling logic aligns with the DM non-zero locations (Star Stencil, width 1). With mpirun -np 2, no errors either. But with mpirun -np 4 or 8, I get errors which say : "Argument out of range/ Inserting a new nonzero at global row/column ... into matrix". 
> 
> I would switch over the logic provided in KSP/ex46.c for filling the Matrix via the MatSetStencil logic but I wanted to know what I was doing wrong in my current code since I've never seen an error that depends on number of mpi processes.
> 
> I'm attaching the code ( which works if the matrix is created without using the DA, i.e. comment out line 159, uncomment 161/162 and I'm doing this on a small grid to catch errors.). 
> 
> Thanks in advance for the help. 
> 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University
> <ex_dmda.c>


From hzhang at mcs.anl.gov  Wed Apr 10 13:27:40 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Wed, 10 Apr 2019 18:27:40 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
Message-ID: <CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>

Myriam,
Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
Hong

On Wed, Apr 10, 2019 at 7:21 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
This looks like it might be noisy data. I'd make sure you run each size on the same set of nodes and you might run each job twice (A,B,A,B) in a job script.

On Wed, Apr 10, 2019 at 8:12 AM Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

Here is the time weak scaling from the same study. The 3.10.2 version seems to be much more stable with regard to the execution time. But not necessarily faster for "large scale" simulations (problem size = 1e8).

I didn't use -mat_freeintermediatedatastructures. I tested it this morning and the solver diverges when using this option (KSPReason -3).

Myriam

Le 04/09/19 ? 17:23, Zhang, Hong a ?crit :
Myriam,
Do you have 'execution time scalability' plot? Did you use '-mat_freeintermediatedatastructures' for PETSc 3.10.2?
We made several computing optimizations on MatPtAP(), which might trade memory for speed. It would be helpful to see a complete comparison.
Hong

On Tue, Apr 9, 2019 at 7:43 AM Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi,

in my first mail, I provided a memory scaling concerning the PETSc
example #42. You'll find attached the main files used (one for PETSc
3.6.4, one for PETSc 3.10.2), and the corresponding memory scaling.

In the main files, I modified the solver/preconditioner, so that it
corresponds to my problem. You'll find the modifications by searching
the keyword "TopBridge". In particular, I use GAMG.

Note that the example is about solving Stokes equation, so using GAMG
may not be adapted. However, the memory gap appears and that's the
point. No matter if the results are correct.

Are these scripts useful for you? Let me know.

Thanks,

Myriam


Le 04/04/19 ? 00:09, Jed Brown a ?crit :
> Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> writes:
>
>> Hi all,
>>
>> for your information, you'll find attached the comparison of the weak
>> memory scalings when using :
>>
>> - PETSc 3.6.4 (reference)
>> - PETSc 3.10.4 without specific options
>> - PETSc 3.10.4 with the three scalability options you mentionned
>>
>> Using the scalability options does improve the memory scaling. However,
>> the 3.6 version still has a better one...
> Yes, this still looks significant.  Is this an effect we can still
> reproduce with a PETSc example and/or using a memory profiler (such as
> massif or gperftools)?  I think it's important for us to narrow down
> what causes this difference (looks like almost 2x on your 1e8 problem
> size) so we can fix.

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--



--
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/c0bee0ca/attachment.html>

From jed at jedbrown.org  Wed Apr 10 15:35:21 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 10 Apr 2019 14:35:21 -0600
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
Message-ID: <877ec17g52.fsf@jedbrown.org>

"Zhang, Hong via petsc-users" <petsc-users at mcs.anl.gov> writes:

> Myriam,
> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.

And yet if turning it on causes divergence, that would imply a bug.
Hong, are you able to reproduce the experiment to see the memory
scaling?

From Mark.Olesen at esi-group.com  Wed Apr 10 16:55:15 2019
From: Mark.Olesen at esi-group.com (Mark Olesen)
Date: Wed, 10 Apr 2019 21:55:15 +0000
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
In-Reply-To: <56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov>
References: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
	<alpine.LFD.2.21.1904100822070.3207@sb>
	<56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov>
Message-ID: <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com>

The paper that Barry mentioned gives some generalities, but probably 
won't help much. There are some PETSc/OpenFOAM interfaces in rheoTool 
that are probably much more helpful.

As Barry also rightly noted, there are some config files in the OpenFOAM 
tree that were put in some time ago for helping with setting up PETSc 
and OpenFOAM. Assuming that you have set the appropriate values in the 
etc/config.sh/petsc file you will be able to use those when using wmake. 
For running you will still need to ensure that the LD_LIBARY_PATH is set 
correctly. For example, what some build scripts exhibit:

wmake(petsc)  :
==> Before running, verify that PETSc libraries can be found

Enable in the OpenFOAM etc/bashrc, define manually or try with the
following (POSIX shell):

     eval $(foamEtcFile -sh -config petsc -- -force)

==


There is currently a PRACE project running via CINECA (Bologna) with 
binding in PETSc as a runtime selectable linear solver in OpenFOAM. This 
is still at the stage of early testing and performance benchmarking.

Cheers,
/mark

On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
> 
>    We don't know much about OpenFoam but
> 
> 1)  if I do a
> 
>    git grep -i petsc
> 
>      in the https://develop.openfoam.com/Development/OpenFOAM-plus.git repository I see various configuration files specifically for PETSc.
> 
> etc/config.csh/petsc  etc/config.sh/petsc wmake/scripts/have_petsc
> 
>      so it appears that OpenFOAM has the tools to be linked against PETSc (to me the documentation on how to use them is rather terse). Are
>      you using these?  If you have trouble with them perhaps you can ask the OpenFOAM user community how to use them.
> 
> 
>    2)  if you are editing the Make/options file directly you can try changing
> 
>       -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> 
>       to
> 
>       -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib   -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> 
> 
> 
>     Note also that simply including petsc.h into the OpenFoam source code and linking against -lpetsc will not immediately allow calling the PETSc solvers from OpenFOAM. One needs to write all the interface code that sets up and calls the PETSc solvers from OpenFOAM. There is a paper https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework that describes at an abstract level how they wrote code that calls the PETSc solvers from OpenFOAM but the source code that actually does the work does not appear to be available.
> 
>     Note that PETSc is now at version 3.11 we recommend working with that version (unless you already have a lot of code that calls PETSc written with a previous version of PETSc, for that we recommend first upgrading to petsc 3.11 and then continuing to add code).
> 
>     Barry
> 
> 
> 
> 
> 
>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Runtime error? You might have to add the path to $PETSC_ARCH/lib in  LD_LIBRARY_PATH env variable
>> or - to your link command. If linux/gcc - the linker option is -Wl,-rpath,$PETSC_ARCH/lib
>>
>> If not - send detail logs.
>>
>> Satish
>>
>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
>>
>>> Hi all,
>>>
>>> I am trying to insert Petsc to OpenFOAM opensource software.
>>> I have been successfully compiling Petsc with an available solver in
>>> OpenFOAM by linking it with the shared library libpetsc.so. However, when I
>>> call the solver to run a test case, I got an error saying that:
>>> "libpetsc.so cannot be found", even though the library still exists in the
>>> $PETSC_ARCH/lib folder.
>>>
>>> I have been struggling for weeks but still, have not been able to figure it
>>> out. Therefore I would be very grateful for any suggestion to solve this
>>> problem.
>>>
>>> Thanks in advance for your time,
>>>
>>> Best regards,
>>>
>>> Vu Do
>>>

From sajidsyed2021 at u.northwestern.edu  Wed Apr 10 18:12:38 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 10 Apr 2019 18:12:38 -0500
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
	<477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>
Message-ID: <CAOGsD9iQtN8jZOjsYF18jebBUCQivHEs8dE7s+mnZHoRW+-Txg@mail.gmail.com>

Thanks a lot for the advice Matt and Barry.

One thing I wanted to confirm is that when I change from using a regular
Vec to a Vec created using DMDACreateGlobalVector, to fill these with data
from hdf5, I have to change the dimensions of hdf5 vectors from
(dim_x*dim_y) to (dim_x,dim_y), right?

Because I see that if I write to hdf5 from a complex vector created using
DMDA, I get a vector that has dimensions (dim_x,dim_y,2) but before I saw
the dimension of the same to be (dim_x*dim_y,2).

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190410/a043d5ed/attachment-0001.html>

From vu.doquochust at gmail.com  Wed Apr 10 19:19:52 2019
From: vu.doquochust at gmail.com (Vu Do Quoc)
Date: Thu, 11 Apr 2019 07:19:52 +0700
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
In-Reply-To: <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com>
References: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
	<alpine.LFD.2.21.1904100822070.3207@sb>
	<56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov>
	<673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com>
Message-ID: <CADAyKL4c16P6APdyzg6DBC9hyq7gUZcdZxvuq+u+Q2_MnPpB9g@mail.gmail.com>

Dear All,

Thanks so much for all of your prompt responses!

By defining the LD_LIBRARY_PATH env variable in .bashrc, I have been able
to call the solver.
However, as Berry mentioned, I cannot directly call for Petsc solver at the
moment. So I guess I would need to create an interface for Petsc and
OpenFOAM.
I have taken a look at rheoTool and it seems like the latest version
provides some really nice reference.

Thanks again for your help.

Best regards,

Vu

On Thu, Apr 11, 2019 at 4:55 AM Mark Olesen <Mark.Olesen at esi-group.com>
wrote:

> The paper that Barry mentioned gives some generalities, but probably
> won't help much. There are some PETSc/OpenFOAM interfaces in rheoTool
> that are probably much more helpful.
>
> As Barry also rightly noted, there are some config files in the OpenFOAM
> tree that were put in some time ago for helping with setting up PETSc
> and OpenFOAM. Assuming that you have set the appropriate values in the
> etc/config.sh/petsc file you will be able to use those when using wmake.
> For running you will still need to ensure that the LD_LIBARY_PATH is set
> correctly. For example, what some build scripts exhibit:
>
> wmake(petsc)  :
> ==> Before running, verify that PETSc libraries can be found
>
> Enable in the OpenFOAM etc/bashrc, define manually or try with the
> following (POSIX shell):
>
>      eval $(foamEtcFile -sh -config petsc -- -force)
>
> ==
>
>
> There is currently a PRACE project running via CINECA (Bologna) with
> binding in PETSc as a runtime selectable linear solver in OpenFOAM. This
> is still at the stage of early testing and performance benchmarking.
>
> Cheers,
> /mark
>
> On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
> >
> >    We don't know much about OpenFoam but
> >
> > 1)  if I do a
> >
> >    git grep -i petsc
> >
> >      in the https://develop.openfoam.com/Development/OpenFOAM-plus.git
> repository I see various configuration files specifically for PETSc.
> >
> > etc/config.csh/petsc  etc/config.sh/petsc wmake/scripts/have_petsc
> >
> >      so it appears that OpenFOAM has the tools to be linked against
> PETSc (to me the documentation on how to use them is rather terse). Are
> >      you using these?  If you have trouble with them perhaps you can ask
> the OpenFOAM user community how to use them.
> >
> >
> >    2)  if you are editing the Make/options file directly you can try
> changing
> >
> >       -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >
> >       to
> >
> >       -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib
>  -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
> >
> >
> >
> >     Note also that simply including petsc.h into the OpenFoam source
> code and linking against -lpetsc will not immediately allow calling the
> PETSc solvers from OpenFOAM. One needs to write all the interface code that
> sets up and calls the PETSc solvers from OpenFOAM. There is a paper
> https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework
> that describes at an abstract level how they wrote code that calls the
> PETSc solvers from OpenFOAM but the source code that actually does the work
> does not appear to be available.
> >
> >     Note that PETSc is now at version 3.11 we recommend working with
> that version (unless you already have a lot of code that calls PETSc
> written with a previous version of PETSc, for that we recommend first
> upgrading to petsc 3.11 and then continuing to add code).
> >
> >     Barry
> >
> >
> >
> >
> >
> >> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>
> >> Runtime error? You might have to add the path to $PETSC_ARCH/lib in
> LD_LIBRARY_PATH env variable
> >> or - to your link command. If linux/gcc - the linker option is
> -Wl,-rpath,$PETSC_ARCH/lib
> >>
> >> If not - send detail logs.
> >>
> >> Satish
> >>
> >> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
> >>
> >>> Hi all,
> >>>
> >>> I am trying to insert Petsc to OpenFOAM opensource software.
> >>> I have been successfully compiling Petsc with an available solver in
> >>> OpenFOAM by linking it with the shared library libpetsc.so. However,
> when I
> >>> call the solver to run a test case, I got an error saying that:
> >>> "libpetsc.so cannot be found", even though the library still exists in
> the
> >>> $PETSC_ARCH/lib folder.
> >>>
> >>> I have been struggling for weeks but still, have not been able to
> figure it
> >>> out. Therefore I would be very grateful for any suggestion to solve
> this
> >>> problem.
> >>>
> >>> Thanks in advance for your time,
> >>>
> >>> Best regards,
> >>>
> >>> Vu Do
> >>>
>


-- 

*Vu Q. Do*------------------------------------------------
*Student of Aeronautical Engineering*
Programme de Formation d'Ing?nieurs d'Excellence au Vietnam *- PFIEV*
School of Transportation Engineering
Hanoi University of Science and Technology
01 Dai Co Viet Avenue, Hanoi, Vietnam
E-mail: vu.doquochust at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/2d414807/attachment.html>

From hzhang at mcs.anl.gov  Wed Apr 10 21:12:56 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 11 Apr 2019 02:12:56 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <877ec17g52.fsf@jedbrown.org>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<8925b24f-62dd-1e45-5658-968491e51205@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
Message-ID: <CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>

Jed:
> Myriam,
> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.

And yet if turning it on causes divergence, that would imply a bug.
Hong, are you able to reproduce the experiment to see the memory
scaling?
I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.
Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/5fc9136d/attachment.html>

From bsmith at mcs.anl.gov  Wed Apr 10 22:09:21 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 11 Apr 2019 03:09:21 +0000
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <CAOGsD9iQtN8jZOjsYF18jebBUCQivHEs8dE7s+mnZHoRW+-Txg@mail.gmail.com>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
	<477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>
	<CAOGsD9iQtN8jZOjsYF18jebBUCQivHEs8dE7s+mnZHoRW+-Txg@mail.gmail.com>
Message-ID: <604E1BB8-F0D5-4A6E-A62B-4AFC6965E2C4@mcs.anl.gov>


   Sajid,

      By default when you save/load vectors from DMDA to HDF5 files it 

1) converts them to the natural ordering in the file (in PETSc programs they are number by process (see the discussions in the users manual about DMDA orderings))

2) it treats it as a 2d array in the HDF5 file (as you discovered). 

This is because it is then easy for other programs like visualization to read the 2d HDF5 arrays and treat them as 2d arrays for further processing (and it doesn't matter how many MPI processes PETSc used the file has the same ordering).

   Barry


> On Apr 10, 2019, at 6:12 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> 
> 
> Thanks a lot for the advice Matt and Barry. 
> 
> One thing I wanted to confirm is that when I change from using a regular Vec to a Vec created using DMDACreateGlobalVector, to fill these with data from hdf5, I have to change the dimensions of hdf5 vectors from (dim_x*dim_y) to (dim_x,dim_y), right? 
> 
> Because I see that if I write to hdf5 from a complex vector created using DMDA, I get a vector that has dimensions (dim_x,dim_y,2) but before I saw the dimension of the same to be (dim_x*dim_y,2). 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University


From jed at jedbrown.org  Wed Apr 10 23:01:13 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 10 Apr 2019 22:01:13 -0600
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<00d471e0-ed2a-51fa-3031-a6b63c3a96e1@idris.fr>
	<CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com>
	<75a2f7b1-9e7a-843f-1a83-efff8e56f797@idris.fr>
	<CAMYG4G==LGGbnuaAzzx8opQWwUck8agtchbLLcfMnPch74Pefg@mail.gmail.com>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
Message-ID: <87zhox5gxi.fsf@jedbrown.org>

"Zhang, Hong" <hzhang at mcs.anl.gov> writes:

> Jed:
>>> Myriam,
>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
>
>> And yet if turning it on causes divergence, that would imply a bug.
>> Hong, are you able to reproduce the experiment to see the memory
>> scaling?
>
> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.

I have now compiled and run her code locally.

Myriam, thanks for your last mail adding configuration and removing the
MemManager.h dependency.  I ran with and without
-mat_freeintermediatedatastructures and don't see a difference in
convergence.  What commands did you run to observe that difference?

From bsmith at mcs.anl.gov  Thu Apr 11 00:48:47 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 11 Apr 2019 05:48:47 +0000
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <CAMYG4GnLVWkQB755o5EMTM4Zts-3MX1A-nEi-tQf-tOqdr--Lw@mail.gmail.com>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
	<CAMYG4GnLVWkQB755o5EMTM4Zts-3MX1A-nEi-tQf-tOqdr--Lw@mail.gmail.com>
Message-ID: <180CC1AB-E940-4F6A-AD55-704C03683748@anl.gov>


  Here is my guess at what is going wrong (and if it is the case we should figure out how to fix it). The KSPSetUp() is triggering the creation of 
work vectors and the type of work vector created at this point is not compatible with the vectors passed into KSPSolve() thus generating an error.

  Meanwhile if you skip calling KSPSetUp(), the KSPSolve() calls KSPSetUp() and by that time KSP knows the vector type (based on the vector passed 
into KSPSolve()) and uses that as a template to generate the work vectors which are now compatible since they were generated using the input vector. It may require using the debugger to track why the wrong type of vector is created for work vectors in KSPSetUp(). It is too late at night for me to try to debug it now.

  Barry


> On Apr 10, 2019, at 11:57 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Wed, Apr 10, 2019 at 12:49 PM Manuel Colera Rico <m.colera at upm.es> wrote:
> Thank you for your answer, Matt. In the MWE example attached before, both Nest vectors (the r.h.s. of the system and the vector of unknowns) are composed of the same number of blocks (2). Indeed, PETSc is able to solve the system if KSPSetUp() is not called, so the system/MatNest/MatVec's must not incompatible at all. Therefore, I wonder if I have missed to called something before this routine or if this is a KSPSetUp's bug.
> 
> I will run it. 
> 
> Of course one can always directly define a single matrix and a single vector, but I find it easier to work with Nest matrices and vectors. Moreover, I think that the moment to use them is from the beginning... once all the code is developed, it is very hard to switch matrices types.
> 
> This is not the case, and its the major reason why I want to remove Vec/MatNest from the public interface. This is a major misconception. You can
> switch matrix types using -mat_type <type>. You can pull out submatrices using MatGetSubmatrix() from MatAIJ just as simply as MatNest.
> If you want to keep assembling submatrices, you can use 
> 
>   https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef
> 
> so that the assembly looks just like it does now for you, but it assembles to an AIJ _or_ a Nest, depending on what you
> choose at the command line. Then you can debug everything as AIJ, and finally switch to Nest if you need it for optimization.
> 
>   Thanks,
> 
>      Matt
> 
> Regards,
> 
> Manuel
> 
> ---
> 
> 
> 
> On 4/10/19 5:41 PM, Matthew Knepley wrote:
>> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> Hello,
>> 
>> I am trying to solve a system whose matrix is of type MatNest. If I 
>> don't use KSPSetUp(), everything is fine. However, if I use that 
>> routine, I get the following error:
>> 
>> 0]PETSC ERROR: --------------------- Error Message 
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Invalid argument
>> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of 
>> blocks.
>> 
>> This seems self-explanatory. Nest vectors must have the same number of blocks to be compatible.
>> 
>> More broadly, there should be no reason to use Nest vectors or matrices. It is an optimization to
>> be used at the very end, only after you have profiled the code and seen that its important. You can
>> do everything you want to do without ever touching Nest, and it looks like the Nest interface is a
>> problem for your code right now.
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>> [0]PETSC ERROR: 
>> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion 
>> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
>> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native" 
>> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 
>> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack 
>> --download--f2cblaslapack --download-mpich --download--hypre 
>> --download-scalapack --download-mumps --download-suitesparse 
>> --download-ptscotch --download-pastix --with-matlab --with-openmp
>> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in 
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>> [0]PETSC ERROR: #2 VecCopy() line 1614 in 
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #5 KSPSolve() line 782 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>> 
>> Please find attached a MWE (it is a slight modification of that of the 
>> post opened by Ce Qin, 
>> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html, 
>> whose answer I have not found).
>> 
>> By the way, with the newest version of PETSc, Eclipse marks as errors 
>> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it 
>> compiles and executes well. Perhaps it is a problem related to Eclipse, 
>> but this did not happen with the older versions of PETSc.
>> 
>> Thanks and regards,
>> 
>> Manuel
>> 
>> ---
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From bsmith at mcs.anl.gov  Thu Apr 11 01:07:55 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 11 Apr 2019 06:07:55 +0000
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
Message-ID: <F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>


   Matt,

     You can't have some functionality in the public API for a library and then criticize someone for using it. I looked at the manual page for MatCreateNest() and it doesn't say in big letters "don't use this", nor does the compiler tell the user "don't use this". Either MatCreateNest() and/or MATNEST is so horrible it needs to be stripped from the face of the earth or it needs to be supported. It can't be in this nebulous state of "yeah it exists but don't ever use it".

   Barry

   In order to support easy testing of MatCreateNest() based user code perhaps we could add a little bit of code that allowed users to call either 
MatCreateNest() or MatNestSetSubMats() but with an option to have it generate a regular AIJ matrix instead. For example -mat_nest_use_aij (and MatNestUseAIJ()) This would allow  users to always write and think about their code as nested matrices but have it "fall back" to AIJ for testing/debugging purposes.


     

> On Apr 10, 2019, at 11:49 AM, Manuel Colera Rico via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thank you for your answer, Matt. In the MWE example attached before, both Nest vectors (the r.h.s. of the system and the vector of unknowns) are composed of the same number of blocks (2). Indeed, PETSc is able to solve the system if KSPSetUp() is not called, so the system/MatNest/MatVec's must not incompatible at all. Therefore, I wonder if I have missed to called something before this routine or if this is a KSPSetUp's bug.
> 
> Of course one can always directly define a single matrix and a single vector, but I find it easier to work with Nest matrices and vectors. Moreover, I think that the moment to use them is from the beginning... once all the code is developed, it is very hard to switch matrices types.
> 
> Regards,
> 
> Manuel
> 
> ---
> 
> 
> 
> On 4/10/19 5:41 PM, Matthew Knepley wrote:
>> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> Hello,
>> 
>> I am trying to solve a system whose matrix is of type MatNest. If I 
>> don't use KSPSetUp(), everything is fine. However, if I use that 
>> routine, I get the following error:
>> 
>> 0]PETSC ERROR: --------------------- Error Message 
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Invalid argument
>> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of 
>> blocks.
>> 
>> This seems self-explanatory. Nest vectors must have the same number of blocks to be compatible.
>> 
>> More broadly, there should be no reason to use Nest vectors or matrices. It is an optimization to
>> be used at the very end, only after you have profiled the code and seen that its important. You can
>> do everything you want to do without ever touching Nest, and it looks like the Nest interface is a
>> problem for your code right now.
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>> [0]PETSC ERROR: 
>> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion 
>> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
>> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native" 
>> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 
>> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack 
>> --download--f2cblaslapack --download-mpich --download--hypre 
>> --download-scalapack --download-mumps --download-suitesparse 
>> --download-ptscotch --download-pastix --with-matlab --with-openmp
>> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in 
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>> [0]PETSC ERROR: #2 VecCopy() line 1614 in 
>> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: #5 KSPSolve() line 782 in 
>> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>> 
>> Please find attached a MWE (it is a slight modification of that of the 
>> post opened by Ce Qin, 
>> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html, 
>> whose answer I have not found).
>> 
>> By the way, with the newest version of PETSc, Eclipse marks as errors 
>> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it 
>> compiles and executes well. Perhaps it is a problem related to Eclipse, 
>> but this did not happen with the older versions of PETSc.
>> 
>> Thanks and regards,
>> 
>> Manuel
>> 
>> ---
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/


From bsmith at mcs.anl.gov  Thu Apr 11 01:36:58 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 11 Apr 2019 06:36:58 +0000
Subject: [petsc-users] Problem coupling Petsc into OpenFOAM
In-Reply-To: <673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com>
References: <CADAyKL4Xni3wC+p3wUQmp5QA7WewL97SzFMTH1PyFygG4-S8nQ@mail.gmail.com>
	<alpine.LFD.2.21.1904100822070.3207@sb>
	<56797615-51EA-4C56-AEB4-F6FEC1949348@anl.gov>
	<673e9f0f-499c-07e3-280f-63ec02f08810@esi-group.com>
Message-ID: <12E2C6D2-8152-42E8-9286-C190FD30AC5D@mcs.anl.gov>


  Mark,

    Thanks for the clarifying email. My google searches didn't locate the rheoTool you mention nor "a PRACE project running via CINECA (Bologna)".

    It would be nice if someday OpenFOAM had (either directly or somehow with the modules directory) an interface to the PETSc solvers. This would allow the use of a variety of other solvers including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM. Unfortunately the PETSc group doesn't have the resources or expertise to develop and support such an interface ourselves. We would, of course, try to answer emails about PETSc usage and bugs for such an interface. 

   Barry

  If OpenFOAM did have such an interface one thing we could provide is the CI infrastructure for tracking changes to PETSc that may effect OpenFOAM. For example we could automatically build OpenFOAM each day with the latest master of PETSc thus immediately detecting changes that effect the interface.




> On Apr 10, 2019, at 4:55 PM, Mark Olesen <Mark.Olesen at esi-group.com> wrote:
> 
> The paper that Barry mentioned gives some generalities, but probably 
> won't help much. There are some PETSc/OpenFOAM interfaces in rheoTool 
> that are probably much more helpful.
> 
> As Barry also rightly noted, there are some config files in the OpenFOAM 
> tree that were put in some time ago for helping with setting up PETSc 
> and OpenFOAM. Assuming that you have set the appropriate values in the 
> etc/config.sh/petsc file you will be able to use those when using wmake. 
> For running you will still need to ensure that the LD_LIBARY_PATH is set 
> correctly. For example, what some build scripts exhibit:
> 
> wmake(petsc)  :
> ==> Before running, verify that PETSc libraries can be found
> 
> Enable in the OpenFOAM etc/bashrc, define manually or try with the
> following (POSIX shell):
> 
>     eval $(foamEtcFile -sh -config petsc -- -force)
> 
> ==
> 
> 
> There is currently a PRACE project running via CINECA (Bologna) with 
> binding in PETSc as a runtime selectable linear solver in OpenFOAM. This 
> is still at the stage of early testing and performance benchmarking.
> 
> Cheers,
> /mark
> 
> On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
>> 
>>   We don't know much about OpenFoam but
>> 
>> 1)  if I do a
>> 
>>   git grep -i petsc
>> 
>>     in the https://develop.openfoam.com/Development/OpenFOAM-plus.git repository I see various configuration files specifically for PETSc.
>> 
>> etc/config.csh/petsc  etc/config.sh/petsc wmake/scripts/have_petsc
>> 
>>     so it appears that OpenFOAM has the tools to be linked against PETSc (to me the documentation on how to use them is rather terse). Are
>>     you using these?  If you have trouble with them perhaps you can ask the OpenFOAM user community how to use them.
>> 
>> 
>>   2)  if you are editing the Make/options file directly you can try changing
>> 
>>      -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>> 
>>      to
>> 
>>      -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib   -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>> 
>> 
>> 
>>    Note also that simply including petsc.h into the OpenFoam source code and linking against -lpetsc will not immediately allow calling the PETSc solvers from OpenFOAM. One needs to write all the interface code that sets up and calls the PETSc solvers from OpenFOAM. There is a paper https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework that describes at an abstract level how they wrote code that calls the PETSc solvers from OpenFOAM but the source code that actually does the work does not appear to be available.
>> 
>>    Note that PETSc is now at version 3.11 we recommend working with that version (unless you already have a lot of code that calls PETSc written with a previous version of PETSc, for that we recommend first upgrading to petsc 3.11 and then continuing to add code).
>> 
>>    Barry
>> 
>> 
>> 
>> 
>> 
>>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Runtime error? You might have to add the path to $PETSC_ARCH/lib in  LD_LIBRARY_PATH env variable
>>> or - to your link command. If linux/gcc - the linker option is -Wl,-rpath,$PETSC_ARCH/lib
>>> 
>>> If not - send detail logs.
>>> 
>>> Satish
>>> 
>>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I am trying to insert Petsc to OpenFOAM opensource software.
>>>> I have been successfully compiling Petsc with an available solver in
>>>> OpenFOAM by linking it with the shared library libpetsc.so. However, when I
>>>> call the solver to run a test case, I got an error saying that:
>>>> "libpetsc.so cannot be found", even though the library still exists in the
>>>> $PETSC_ARCH/lib folder.
>>>> 
>>>> I have been struggling for weeks but still, have not been able to figure it
>>>> out. Therefore I would be very grateful for any suggestion to solve this
>>>> problem.
>>>> 
>>>> Thanks in advance for your time,
>>>> 
>>>> Best regards,
>>>> 
>>>> Vu Do
>>>> 


From knepley at gmail.com  Thu Apr 11 04:30:44 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Apr 2019 05:30:44 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
	<F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>
Message-ID: <CAMYG4G=X6ZRkQjihAjBmRMqa-JPYDJDQ66RWAJq65+mLtH8zXw@mail.gmail.com>

On Thu, Apr 11, 2019 at 2:07 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    Matt,
>
>      You can't have some functionality in the public API for a library and
> then criticize someone for using it. I looked at the manual page for
> MatCreateNest() and it doesn't say in big letters "don't use this", nor
> does the compiler tell the user "don't use this". Either MatCreateNest()
> and/or MATNEST is so horrible it needs to be stripped from the face of the
> earth or it needs to be supported. It can't be in this nebulous state of
> "yeah it exists but don't ever use it".
>

I agree this is a documentation problem. That is why I try to provide
complete explanations for my recommendations.


>    Barry
>
>    In order to support easy testing of MatCreateNest() based user code
> perhaps we could add a little bit of code that allowed users to call either
> MatCreateNest() or MatNestSetSubMats() but with an option to have it
> generate a regular AIJ matrix instead. For example -mat_nest_use_aij (and
> MatNestUseAIJ()) This would allow  users to always write and think about
> their code as nested matrices but have it "fall back" to AIJ for
> testing/debugging purposes.
>

I think we should remove MatCreateNest() completely. We should have 1
mechanism for getting submatrices for creation, not 2, so we retain
MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
back an actually submatrix, otherwise it gives a view. This would do
everything that we currently do without this horrible MatNest interface
bubbling to the top.

   Matt


> > On Apr 10, 2019, at 11:49 AM, Manuel Colera Rico via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Thank you for your answer, Matt. In the MWE example attached before,
> both Nest vectors (the r.h.s. of the system and the vector of unknowns) are
> composed of the same number of blocks (2). Indeed, PETSc is able to solve
> the system if KSPSetUp() is not called, so the system/MatNest/MatVec's must
> not incompatible at all. Therefore, I wonder if I have missed to called
> something before this routine or if this is a KSPSetUp's bug.
> >
> > Of course one can always directly define a single matrix and a single
> vector, but I find it easier to work with Nest matrices and vectors.
> Moreover, I think that the moment to use them is from the beginning... once
> all the code is developed, it is very hard to switch matrices types.
> >
> > Regards,
> >
> > Manuel
> >
> > ---
> >
> >
> >
> > On 4/10/19 5:41 PM, Matthew Knepley wrote:
> >> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >> Hello,
> >>
> >> I am trying to solve a system whose matrix is of type MatNest. If I
> >> don't use KSPSetUp(), everything is fine. However, if I use that
> >> routine, I get the following error:
> >>
> >> 0]PETSC ERROR: --------------------- Error Message
> >> --------------------------------------------------------------
> >> [0]PETSC ERROR: Invalid argument
> >> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers of
> >> blocks.
> >>
> >> This seems self-explanatory. Nest vectors must have the same number of
> blocks to be compatible.
> >>
> >> More broadly, there should be no reason to use Nest vectors or
> matrices. It is an optimization to
> >> be used at the very end, only after you have profiled the code and seen
> that its important. You can
> >> do everything you want to do without ever touching Nest, and it looks
> like the Nest interface is a
> >> problem for your code right now.
> >>
> >>   Thanks,
> >>
> >>     Matt
> >>
> >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> for trouble shooting.
> >> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
> >> [0]PETSC ERROR:
> >>
> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>
> >> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> >> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
> >> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
> >> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack
> >> --download--f2cblaslapack --download-mpich --download--hypre
> >> --download-scalapack --download-mumps --download-suitesparse
> >> --download-ptscotch --download-pastix --with-matlab --with-openmp
> >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
> >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
> >> [0]PETSC ERROR: #2 VecCopy() line 1614 in
> >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
> >> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
> >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
> >> [0]PETSC ERROR: #5 KSPSolve() line 782 in
> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
> >> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
> >>
> >> Please find attached a MWE (it is a slight modification of that of the
> >> post opened by Ce Qin,
> >>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>
> >> whose answer I have not found).
> >>
> >> By the way, with the newest version of PETSc, Eclipse marks as errors
> >> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
> >> compiles and executes well. Perhaps it is a problem related to Eclipse,
> >> but this did not happen with the older versions of PETSc.
> >>
> >> Thanks and regards,
> >>
> >> Manuel
> >>
> >> ---
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> -- Norbert Wiener
> >>
> >> https://www.cse.buffalo.edu/~knepley/
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/5b9a713f/attachment-0001.html>

From m.colera at upm.es  Thu Apr 11 04:57:33 2019
From: m.colera at upm.es (Manuel Colera Rico)
Date: Thu, 11 Apr 2019 11:57:33 +0200
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <CAMYG4G=X6ZRkQjihAjBmRMqa-JPYDJDQ66RWAJq65+mLtH8zXw@mail.gmail.com>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
	<F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>
	<CAMYG4G=X6ZRkQjihAjBmRMqa-JPYDJDQ66RWAJq65+mLtH8zXw@mail.gmail.com>
Message-ID: <e9840e8a-21a8-4f06-1d95-ba681139f957@upm.es>

MatCreateLocalRef() seems to be useful to extract submatrices from a 
global matrix. But how to construct A from the beginning from its 
submatrices? I.e., with MatNest, I can just do:

 ??? Mat A;

 ??? Mat matA[] = {A00, A01, NULL, A11};

 ??? MatCreateNest(PETSC_COMM_SELF, 2, NULL, 2, NULL, matA, &A);

without worrying about memory preallocations for A. Can I do the same 
with MatCreateLocalRef(), or do I first have to preallocate memory for A?

Manuel

---

On 4/11/19 11:30 AM, Matthew Knepley wrote:

> On Thu, Apr 11, 2019 at 2:07 AM Smith, Barry F. <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>     ? ?Matt,
>
>     ? ? ?You can't have some functionality in the public API for a
>     library and then criticize someone for using it. I looked at the
>     manual page for MatCreateNest() and it doesn't say in big letters
>     "don't use this", nor does the compiler tell the user "don't use
>     this". Either MatCreateNest() and/or MATNEST is so horrible it
>     needs to be stripped from the face of the earth or it needs to be
>     supported. It can't be in this nebulous state of "yeah it exists
>     but don't ever use it".
>
>
> I agree this is a documentation problem. That is why I try to provide 
> complete explanations for my recommendations.
>
>     ? ?Barry
>
>     ? ?In order to support easy testing of MatCreateNest() based user
>     code perhaps we could add a little bit of code that allowed users
>     to call either
>     MatCreateNest() or MatNestSetSubMats() but with an option to have
>     it generate a regular AIJ matrix instead. For example
>     -mat_nest_use_aij (and MatNestUseAIJ()) This would allow users to
>     always write and think about their code as nested matrices but
>     have it "fall back" to AIJ for testing/debugging purposes.
>
>
> I think we should remove MatCreateNest() completely. We should have 1 
> mechanism for getting submatrices for creation, not 2, so we retain
> MatCreateLocalRef(). Then if -mat_type nest was specified, it gives 
> you back an actually submatrix, otherwise it gives a view. This would do
> everything that we currently do without this horrible MatNest 
> interface bubbling to the top.
>
> ? ?Matt
>
>     > On Apr 10, 2019, at 11:49 AM, Manuel Colera Rico via petsc-users
>     <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>     >
>     > Thank you for your answer, Matt. In the MWE example attached
>     before, both Nest vectors (the r.h.s. of the system and the vector
>     of unknowns) are composed of the same number of blocks (2).
>     Indeed, PETSc is able to solve the system if KSPSetUp() is not
>     called, so the system/MatNest/MatVec's must not incompatible at
>     all. Therefore, I wonder if I have missed to called something
>     before this routine or if this is a KSPSetUp's bug.
>     >
>     > Of course one can always directly define a single matrix and a
>     single vector, but I find it easier to work with Nest matrices and
>     vectors. Moreover, I think that the moment to use them is from the
>     beginning... once all the code is developed, it is very hard to
>     switch matrices types.
>     >
>     > Regards,
>     >
>     > Manuel
>     >
>     > ---
>     >
>     >
>     >
>     > On 4/10/19 5:41 PM, Matthew Knepley wrote:
>     >> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via
>     petsc-users <petsc-users at mcs.anl.gov
>     <mailto:petsc-users at mcs.anl.gov>> wrote:
>     >> Hello,
>     >>
>     >> I am trying to solve a system whose matrix is of type MatNest.
>     If I
>     >> don't use KSPSetUp(), everything is fine. However, if I use that
>     >> routine, I get the following error:
>     >>
>     >> 0]PETSC ERROR: --------------------- Error Message
>     >> --------------------------------------------------------------
>     >> [0]PETSC ERROR: Invalid argument
>     >> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different
>     numbers of
>     >> blocks.
>     >>
>     >> This seems self-explanatory. Nest vectors must have the same
>     number of blocks to be compatible.
>     >>
>     >> More broadly, there should be no reason to use Nest vectors or
>     matrices. It is an optimization to
>     >> be used at the very end, only after you have profiled the code
>     and seen that its important. You can
>     >> do everything you want to do without ever touching Nest, and it
>     looks like the Nest interface is a
>     >> problem for your code right now.
>     >>
>     >>? ?Thanks,
>     >>
>     >>? ? ?Matt
>     >>
>     >> [0]PETSC ERROR: See
>     http://www.mcs.anl.gov/petsc/documentation/faq.html
>     >> for trouble shooting.
>     >> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>     >> [0]PETSC ERROR:
>     >>
>     /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>
>     >> on a mcr_20190405 named mancolric by Unknown Wed Apr 10
>     17:20:16 2019
>     >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>     >> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
>     >> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>     >> -march=native -mtune=native" --with-debugging=0
>     --download-fblaslapack
>     >> --download--f2cblaslapack --download-mpich --download--hypre
>     >> --download-scalapack --download-mumps --download-suitesparse
>     >> --download-ptscotch --download-pastix --with-matlab --with-openmp
>     >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
>     >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>     >> [0]PETSC ERROR: #2 VecCopy() line 1614 in
>     >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>     >> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
>     >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>     >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
>     >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>     >> [0]PETSC ERROR: #5 KSPSolve() line 782 in
>     >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>     >> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>     >>
>     >> Please find attached a MWE (it is a slight modification of that
>     of the
>     >> post opened by Ce Qin,
>     >>
>     https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>
>     >> whose answer I have not found).
>     >>
>     >> By the way, with the newest version of PETSc, Eclipse marks as
>     errors
>     >> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
>     >> compiles and executes well. Perhaps it is a problem related to
>     Eclipse,
>     >> but this did not happen with the older versions of PETSc.
>     >>
>     >> Thanks and regards,
>     >>
>     >> Manuel
>     >>
>     >> ---
>     >>
>     >>
>     >>
>     >> --
>     >> What most experimenters take for granted before they begin
>     their experiments is infinitely more interesting than any results
>     to which their experiments lead.
>     >> -- Norbert Wiener
>     >>
>     >> https://www.cse.buffalo.edu/~knepley/
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/da2c0993/attachment.html>

From knepley at gmail.com  Thu Apr 11 05:07:23 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Apr 2019 06:07:23 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <e9840e8a-21a8-4f06-1d95-ba681139f957@upm.es>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
	<F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>
	<CAMYG4G=X6ZRkQjihAjBmRMqa-JPYDJDQ66RWAJq65+mLtH8zXw@mail.gmail.com>
	<e9840e8a-21a8-4f06-1d95-ba681139f957@upm.es>
Message-ID: <CAMYG4Gnc66=netezX4sJWD-s4PceZu2B3UUNghVMOEPqhWTwXw@mail.gmail.com>

On Thu, Apr 11, 2019 at 5:57 AM Manuel Colera Rico <m.colera at upm.es> wrote:

> MatCreateLocalRef() seems to be useful to extract submatrices from a
> global matrix. But how to construct A from the beginning from its
> submatrices? I.e., with MatNest, I can just do:
>
>     Mat A;
>
>     Mat matA[] = {A00, A01, NULL, A11};
>
>     MatCreateNest(PETSC_COMM_SELF, 2, NULL, 2, NULL, matA, &A);
>
> without worrying about memory preallocations for A. Can I do the same with
> MatCreateLocalRef(), or do I first have to preallocate memory for A?
>
> It is a good point. You would have to preallocate the large matrix.
However, the preallocation is just the sum for the
preallocations of the small matrices, which you are already specifying. It
would be no problem to give a similar
LocalRef interface to preallocation to do that sum automatically for the
user.

Moreover, we now have a MatPreallocator interface. You can run your matrix
assembly twice. Once with the
MatPreallocator matrix, which can then preallocate you matrix exactly, and
then again with the real matrix. This
is more expensive of course, but in many applications, this expense is
negligible compared to the solver.

  Thanks,

     Matt

> Manuel
>
> ---
>
> On 4/11/19 11:30 AM, Matthew Knepley wrote:
>
> On Thu, Apr 11, 2019 at 2:07 AM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>>    Matt,
>>
>>      You can't have some functionality in the public API for a library
>> and then criticize someone for using it. I looked at the manual page for
>> MatCreateNest() and it doesn't say in big letters "don't use this", nor
>> does the compiler tell the user "don't use this". Either MatCreateNest()
>> and/or MATNEST is so horrible it needs to be stripped from the face of the
>> earth or it needs to be supported. It can't be in this nebulous state of
>> "yeah it exists but don't ever use it".
>>
>
> I agree this is a documentation problem. That is why I try to provide
> complete explanations for my recommendations.
>
>
>>    Barry
>>
>>    In order to support easy testing of MatCreateNest() based user code
>> perhaps we could add a little bit of code that allowed users to call either
>> MatCreateNest() or MatNestSetSubMats() but with an option to have it
>> generate a regular AIJ matrix instead. For example -mat_nest_use_aij (and
>> MatNestUseAIJ()) This would allow  users to always write and think about
>> their code as nested matrices but have it "fall back" to AIJ for
>> testing/debugging purposes.
>>
>
> I think we should remove MatCreateNest() completely. We should have 1
> mechanism for getting submatrices for creation, not 2, so we retain
> MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
> back an actually submatrix, otherwise it gives a view. This would do
> everything that we currently do without this horrible MatNest interface
> bubbling to the top.
>
>    Matt
>
>
>> > On Apr 10, 2019, at 11:49 AM, Manuel Colera Rico via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Thank you for your answer, Matt. In the MWE example attached before,
>> both Nest vectors (the r.h.s. of the system and the vector of unknowns) are
>> composed of the same number of blocks (2). Indeed, PETSc is able to solve
>> the system if KSPSetUp() is not called, so the system/MatNest/MatVec's must
>> not incompatible at all. Therefore, I wonder if I have missed to called
>> something before this routine or if this is a KSPSetUp's bug.
>> >
>> > Of course one can always directly define a single matrix and a single
>> vector, but I find it easier to work with Nest matrices and vectors.
>> Moreover, I think that the moment to use them is from the beginning... once
>> all the code is developed, it is very hard to switch matrices types.
>> >
>> > Regards,
>> >
>> > Manuel
>> >
>> > ---
>> >
>> >
>> >
>> > On 4/10/19 5:41 PM, Matthew Knepley wrote:
>> >> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >> Hello,
>> >>
>> >> I am trying to solve a system whose matrix is of type MatNest. If I
>> >> don't use KSPSetUp(), everything is fine. However, if I use that
>> >> routine, I get the following error:
>> >>
>> >> 0]PETSC ERROR: --------------------- Error Message
>> >> --------------------------------------------------------------
>> >> [0]PETSC ERROR: Invalid argument
>> >> [0]PETSC ERROR: Nest vector arguments 1 and 2 have different numbers
>> of
>> >> blocks.
>> >>
>> >> This seems self-explanatory. Nest vectors must have the same number of
>> blocks to be compatible.
>> >>
>> >> More broadly, there should be no reason to use Nest vectors or
>> matrices. It is an optimization to
>> >> be used at the very end, only after you have profiled the code and
>> seen that its important. You can
>> >> do everything you want to do without ever touching Nest, and it looks
>> like the Nest interface is a
>> >> problem for your code right now.
>> >>
>> >>   Thanks,
>> >>
>> >>     Matt
>> >>
>> >> [0]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>> >> for trouble shooting.
>> >> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>> >> [0]PETSC ERROR:
>> >>
>> /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>>
>> >> on a mcr_20190405 named mancolric by Unknown Wed Apr 10 17:20:16 2019
>> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> >> --with-fc=gfortran COPTFLAGS="-O3 -march=native -mtune=native"
>> >> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>> >> -march=native -mtune=native" --with-debugging=0 --download-fblaslapack
>> >> --download--f2cblaslapack --download-mpich --download--hypre
>> >> --download-scalapack --download-mumps --download-suitesparse
>> >> --download-ptscotch --download-pastix --with-matlab --with-openmp
>> >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
>> >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>> >> [0]PETSC ERROR: #2 VecCopy() line 1614 in
>> >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>> >> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
>> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>> >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
>> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>> >> [0]PETSC ERROR: #5 KSPSolve() line 782 in
>> >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>> >> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>> >>
>> >> Please find attached a MWE (it is a slight modification of that of the
>> >> post opened by Ce Qin,
>> >>
>> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>>
>> >> whose answer I have not found).
>> >>
>> >> By the way, with the newest version of PETSc, Eclipse marks as errors
>> >> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,... although it
>> >> compiles and executes well. Perhaps it is a problem related to
>> Eclipse,
>> >> but this did not happen with the older versions of PETSc.
>> >>
>> >> Thanks and regards,
>> >>
>> >> Manuel
>> >>
>> >> ---
>> >>
>> >>
>> >>
>> >> --
>> >> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> >> -- Norbert Wiener
>> >>
>> >> https://www.cse.buffalo.edu/~knepley/
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/9008045e/attachment-0001.html>

From m.colera at upm.es  Thu Apr 11 05:13:33 2019
From: m.colera at upm.es (Manuel Colera Rico)
Date: Thu, 11 Apr 2019 12:13:33 +0200
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <CAMYG4Gnc66=netezX4sJWD-s4PceZu2B3UUNghVMOEPqhWTwXw@mail.gmail.com>
References: <a704f6c0-92c3-ebcb-eee9-617ab8d5f812@upm.es>
	<CAMYG4Gmnf2YdHBoXNo68sbcrBiF+s9i3=JU1b1kgUMSAwxUMyQ@mail.gmail.com>
	<da0756b6-10a1-952f-cb00-dd5d90aef7be@upm.es>
	<F680D12C-CF9C-4C0C-A871-65F7A757BCFE@anl.gov>
	<CAMYG4G=X6ZRkQjihAjBmRMqa-JPYDJDQ66RWAJq65+mLtH8zXw@mail.gmail.com>
	<e9840e8a-21a8-4f06-1d95-ba681139f957@upm.es>
	<CAMYG4Gnc66=netezX4sJWD-s4PceZu2B3UUNghVMOEPqhWTwXw@mail.gmail.com>
Message-ID: <c1d518fd-0aa9-f341-f5a9-3aeb7ff7bd18@upm.es>

OK, many thanks,

Manuel

---

On 4/11/19 12:07 PM, Matthew Knepley wrote:
> On Thu, Apr 11, 2019 at 5:57 AM Manuel Colera Rico <m.colera at upm.es 
> <mailto:m.colera at upm.es>> wrote:
>
>     MatCreateLocalRef() seems to be useful to extract submatrices from
>     a global matrix. But how to construct A from the beginning from
>     its submatrices? I.e., with MatNest, I can just do:
>
>     ??? Mat A;
>
>     ??? Mat matA[] = {A00, A01, NULL, A11};
>
>     ??? MatCreateNest(PETSC_COMM_SELF, 2, NULL, 2, NULL, matA, &A);
>
>     without worrying about memory preallocations for A. Can I do the
>     same with MatCreateLocalRef(), or do I first have to preallocate
>     memory for A?
>
> It is a good point. You would have to preallocate the large matrix. 
> However, the preallocation is just the sum for the
> preallocations of the small matrices, which you are already 
> specifying. It would be no problem to give a similar
> LocalRef interface to preallocation to do that sum automatically for 
> the user.
>
> Moreover, we now have a MatPreallocator interface. You can run your 
> matrix assembly twice. Once with the
> MatPreallocator matrix, which can then preallocate you matrix exactly, 
> and then again with the real matrix. This
> is more expensive of course, but in many applications, this expense is 
> negligible compared to the solver.
>
> ? Thanks,
>
> ? ? ?Matt
>
>     Manuel
>
>     ---
>
>     On 4/11/19 11:30 AM, Matthew Knepley wrote:
>
>>     On Thu, Apr 11, 2019 at 2:07 AM Smith, Barry F.
>>     <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>
>>
>>         ? ?Matt,
>>
>>         ? ? ?You can't have some functionality in the public API for
>>         a library and then criticize someone for using it. I looked
>>         at the manual page for MatCreateNest() and it doesn't say in
>>         big letters "don't use this", nor does the compiler tell the
>>         user "don't use this". Either MatCreateNest() and/or MATNEST
>>         is so horrible it needs to be stripped from the face of the
>>         earth or it needs to be supported. It can't be in this
>>         nebulous state of "yeah it exists but don't ever use it".
>>
>>
>>     I agree this is a documentation problem. That is why I try to
>>     provide complete explanations for my recommendations.
>>
>>         ? ?Barry
>>
>>         ? ?In order to support easy testing of MatCreateNest() based
>>         user code perhaps we could add a little bit of code that
>>         allowed users to call either
>>         MatCreateNest() or MatNestSetSubMats() but with an option to
>>         have it generate a regular AIJ matrix instead. For example
>>         -mat_nest_use_aij (and MatNestUseAIJ()) This would allow?
>>         users to always write and think about their code as nested
>>         matrices but have it "fall back" to AIJ for testing/debugging
>>         purposes.
>>
>>
>>     I think we should remove MatCreateNest() completely. We should
>>     have 1 mechanism for getting submatrices for creation, not 2, so
>>     we retain
>>     MatCreateLocalRef(). Then if -mat_type nest was specified, it
>>     gives you back an actually submatrix, otherwise it gives a view.
>>     This would do
>>     everything that we currently do without this horrible MatNest
>>     interface bubbling to the top.
>>
>>     ? ?Matt
>>
>>         > On Apr 10, 2019, at 11:49 AM, Manuel Colera Rico via
>>         petsc-users <petsc-users at mcs.anl.gov
>>         <mailto:petsc-users at mcs.anl.gov>> wrote:
>>         >
>>         > Thank you for your answer, Matt. In the MWE example
>>         attached before, both Nest vectors (the r.h.s. of the system
>>         and the vector of unknowns) are composed of the same number
>>         of blocks (2). Indeed, PETSc is able to solve the system if
>>         KSPSetUp() is not called, so the system/MatNest/MatVec's must
>>         not incompatible at all. Therefore, I wonder if I have missed
>>         to called something before this routine or if this is a
>>         KSPSetUp's bug.
>>         >
>>         > Of course one can always directly define a single matrix
>>         and a single vector, but I find it easier to work with Nest
>>         matrices and vectors. Moreover, I think that the moment to
>>         use them is from the beginning... once all the code is
>>         developed, it is very hard to switch matrices types.
>>         >
>>         > Regards,
>>         >
>>         > Manuel
>>         >
>>         > ---
>>         >
>>         >
>>         >
>>         > On 4/10/19 5:41 PM, Matthew Knepley wrote:
>>         >> On Wed, Apr 10, 2019 at 11:29 AM Manuel Colera Rico via
>>         petsc-users <petsc-users at mcs.anl.gov
>>         <mailto:petsc-users at mcs.anl.gov>> wrote:
>>         >> Hello,
>>         >>
>>         >> I am trying to solve a system whose matrix is of type
>>         MatNest. If I
>>         >> don't use KSPSetUp(), everything is fine. However, if I
>>         use that
>>         >> routine, I get the following error:
>>         >>
>>         >> 0]PETSC ERROR: --------------------- Error Message
>>         >> --------------------------------------------------------------
>>         >> [0]PETSC ERROR: Invalid argument
>>         >> [0]PETSC ERROR: Nest vector arguments 1 and 2 have
>>         different numbers of
>>         >> blocks.
>>         >>
>>         >> This seems self-explanatory. Nest vectors must have the
>>         same number of blocks to be compatible.
>>         >>
>>         >> More broadly, there should be no reason to use Nest
>>         vectors or matrices. It is an optimization to
>>         >> be used at the very end, only after you have profiled the
>>         code and seen that its important. You can
>>         >> do everything you want to do without ever touching Nest,
>>         and it looks like the Nest interface is a
>>         >> problem for your code right now.
>>         >>
>>         >>? ?Thanks,
>>         >>
>>         >>? ? ?Matt
>>         >>
>>         >> [0]PETSC ERROR: See
>>         http://www.mcs.anl.gov/petsc/documentation/faq.html
>>         >> for trouble shooting.
>>         >> [0]PETSC ERROR: Petsc Release Version 3.11.0, unknown
>>         >> [0]PETSC ERROR:
>>         >>
>>         /home/manu/Documents/FEM-fluids/C-codes/CLG2-ConvectionDiffusion/Debug/CLG2-ConvectionDiffusion
>>
>>         >> on a mcr_20190405 named mancolric by Unknown Wed Apr 10
>>         17:20:16 2019
>>         >> [0]PETSC ERROR: Configure options --with-cc=gcc
>>         --with-cxx=g++
>>         >> --with-fc=gfortran COPTFLAGS="-O3 -march=native
>>         -mtune=native"
>>         >> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>>         >> -march=native -mtune=native" --with-debugging=0
>>         --download-fblaslapack
>>         >> --download--f2cblaslapack --download-mpich --download--hypre
>>         >> --download-scalapack --download-mumps --download-suitesparse
>>         >> --download-ptscotch --download-pastix --with-matlab
>>         --with-openmp
>>         >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in
>>         >>
>>         /opt/PETSc_library/petsc-3.11.0/src/vec/vec/impls/nest/vecnest.c
>>         >> [0]PETSC ERROR: #2 VecCopy() line 1614 in
>>         >> /opt/PETSc_library/petsc-3.11.0/src/vec/vec/interface/vector.c
>>         >> [0]PETSC ERROR: #3 KSPInitialResidual() line 63 in
>>         >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itres.c
>>         >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in
>>         >>
>>         /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/impls/gmres/gmres.c
>>         >> [0]PETSC ERROR: #5 KSPSolve() line 782 in
>>         >> /opt/PETSc_library/petsc-3.11.0/src/ksp/ksp/interface/itfunc.c
>>         >> [0]PETSC ERROR: #6 mwe() line 55 in ../Tests/tests.c
>>         >>
>>         >> Please find attached a MWE (it is a slight modification of
>>         that of the
>>         >> post opened by Ce Qin,
>>         >>
>>         https://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024230.html,
>>
>>         >> whose answer I have not found).
>>         >>
>>         >> By the way, with the newest version of PETSc, Eclipse
>>         marks as errors
>>         >> the commands PetscFree, CHKERRQ, PETSC_COMM_SELF,...
>>         although it
>>         >> compiles and executes well. Perhaps it is a problem
>>         related to Eclipse,
>>         >> but this did not happen with the older versions of PETSc.
>>         >>
>>         >> Thanks and regards,
>>         >>
>>         >> Manuel
>>         >>
>>         >> ---
>>         >>
>>         >>
>>         >>
>>         >> --
>>         >> What most experimenters take for granted before they begin
>>         their experiments is infinitely more interesting than any
>>         results to which their experiments lead.
>>         >> -- Norbert Wiener
>>         >>
>>         >> https://www.cse.buffalo.edu/~knepley/
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/49aa75a8/attachment.html>

From C.Klaij at marin.nl  Thu Apr 11 06:50:59 2019
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Thu, 11 Apr 2019 11:50:59 +0000
Subject: [petsc-users] Error with KSPSetUp and MatNest
Message-ID: <1554983460037.66141@marin.nl>

Matt,

As a happy MATNEST user, I got a bit worried when you wrote "we
should remove MatCreateNest() completely". This happened last
time I tried to use AIJ instead:

https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-August/029973.html

Has this problem been fixed in the meantime?

Chris

> I think we should remove MatCreateNest() completely. We should have 1
> mechanism for getting submatrices for creation, not 2, so we retain
> MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
> back an actually submatrix, otherwise it gives a view. This would do
> everything that we currently do without this horrible MatNest interface
> bubbling to the top.
>
>    Matt
>


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/42nd-FPSO-JIP-Week-April-812-Singapore.htm


From knepley at gmail.com  Thu Apr 11 07:16:57 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Apr 2019 08:16:57 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <1554983460037.66141@marin.nl>
References: <1554983460037.66141@marin.nl>
Message-ID: <CAMYG4G=GchdSCgrY8XM7_Nin-5Y+5df5UggVg_24F_roQ6ZYBA@mail.gmail.com>

On Thu, Apr 11, 2019 at 7:51 AM Klaij, Christiaan via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Matt,
>
> As a happy MATNEST user, I got a bit worried when you wrote "we
> should remove MatCreateNest() completely".


This would not remove any of the Nest functionality, just the direct
interface to it, which is the problem.


> This happened last
> time I tried to use AIJ instead:
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-August/029973.html
>
> Has this problem been fixed in the meantime?
>

This looks like a problem with the Nest implementation, not AIJ? Maybe I am
misunderstanding it.

   Thanks,

     Matt


> Chris
>
> > I think we should remove MatCreateNest() completely. We should have 1
> > mechanism for getting submatrices for creation, not 2, so we retain
> > MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
> > back an actually submatrix, otherwise it gives a view. This would do
> > everything that we currently do without this horrible MatNest interface
> > bubbling to the top.
> >
> >    Matt
> >
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>
> MARIN news:
> http://www.marin.nl/web/News/News-items/42nd-FPSO-JIP-Week-April-812-Singapore.htm
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/8d989374/attachment-0001.html>

From C.Klaij at marin.nl  Thu Apr 11 08:58:22 2019
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Thu, 11 Apr 2019 13:58:22 +0000
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <CAMYG4G=GchdSCgrY8XM7_Nin-5Y+5df5UggVg_24F_roQ6ZYBA@mail.gmail.com>
References: <1554983460037.66141@marin.nl>,
	<CAMYG4G=GchdSCgrY8XM7_Nin-5Y+5df5UggVg_24F_roQ6ZYBA@mail.gmail.com>
Message-ID: <1554991102771.53007@marin.nl>

Just like Manuel Colera Rico, I would like to begin with existing
(sub)matrices and put them directly into a matnest.

You seem to have understood that problem from the archive 2.5
years ago... If my memory is correct, it was an attempt to
create a mat and switch between -mat_type nest and aij on the
command line.


Chris

dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl<mailto:C.Klaij at marin.nl> | www.marin.nl<http://www.marin.nl>

[LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] <http://www.youtube.com/marinmultimedia>  [Twitter] <https://twitter.com/MARIN_nieuws>  [Facebook] <https://www.facebook.com/marin.wageningen>
MARIN news: First autonomous manoeuvring vessel trials held on North Sea<http://www.marin.nl/web/News/News-items/First-autonomous-manoeuvring-vessel-trials-held-on-North-Sea.htm>

________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, April 11, 2019 2:16 PM
To: Klaij, Christiaan
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Error with KSPSetUp and MatNest

On Thu, Apr 11, 2019 at 7:51 AM Klaij, Christiaan via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Matt,

As a happy MATNEST user, I got a bit worried when you wrote "we
should remove MatCreateNest() completely".

This would not remove any of the Nest functionality, just the direct interface to it, which is the problem.

This happened last
time I tried to use AIJ instead:

https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-August/029973.html

Has this problem been fixed in the meantime?

This looks like a problem with the Nest implementation, not AIJ? Maybe I am misunderstanding it.

   Thanks,

     Matt

Chris

> I think we should remove MatCreateNest() completely. We should have 1
> mechanism for getting submatrices for creation, not 2, so we retain
> MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
> back an actually submatrix, otherwise it gives a view. This would do
> everything that we currently do without this horrible MatNest interface
> bubbling to the top.
>
>    Matt
>


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl> | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/42nd-FPSO-JIP-Week-April-812-Singapore.htm



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>



Help us improve the spam filter. If this message contains SPAM, click here<https://www.mailcontrol.com/sr/VaxGuPX0yKPGX2PQPOmvUqEZpHzWxmiI7TcWSqBM5JDxEDemOSATW7gNYv05LkIR8iuercSaQ0xkxH4nYzTQYQ==> to report. Thank you, MARIN Support Group


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/e1dda512/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagea896fb.PNG
Type: image/png
Size: 293 bytes
Desc: imagea896fb.PNG
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/e1dda512/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image0539c8.PNG
Type: image/png
Size: 331 bytes
Desc: image0539c8.PNG
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/e1dda512/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagefee82c.PNG
Type: image/png
Size: 333 bytes
Desc: imagefee82c.PNG
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/e1dda512/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image188b81.PNG
Type: image/png
Size: 253 bytes
Desc: image188b81.PNG
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/e1dda512/attachment-0003.png>

From knepley at gmail.com  Thu Apr 11 09:00:23 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Apr 2019 10:00:23 -0400
Subject: [petsc-users] Error with KSPSetUp and MatNest
In-Reply-To: <1554991102771.53007@marin.nl>
References: <1554983460037.66141@marin.nl>
	<CAMYG4G=GchdSCgrY8XM7_Nin-5Y+5df5UggVg_24F_roQ6ZYBA@mail.gmail.com>
	<1554991102771.53007@marin.nl>
Message-ID: <CAMYG4Gk_KHwBZOB-K-=9qL7Nz5sEqd+k7UB0QHCY84+qKgsKvw@mail.gmail.com>

On Thu, Apr 11, 2019 at 9:58 AM Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> Just like Manuel Colera Rico, I would like to begin with existing
> (sub)matrices and put them directly into a matnest.
>
> You seem to have understood that problem from the archive 2.5
> years ago... If my memory is correct, it was an attempt to
> create a mat and switch between -mat_type nest and aij on the
> command line.
>
> Okay, I will make an example once I get back from my trip and fix anything
that does not work.

  Thanks,

     Matt

> Chris
>
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl | www.marin.nl
>
> [image: LinkedIn] <https://www.linkedin.com/company/marin> [image:
> YouTube] <http://www.youtube.com/marinmultimedia> [image: Twitter]
> <https://twitter.com/MARIN_nieuws> [image: Facebook]
> <https://www.facebook.com/marin.wageningen>
> MARIN news: First autonomous manoeuvring vessel trials held on North Sea
> <http://www.marin.nl/web/News/News-items/First-autonomous-manoeuvring-vessel-trials-held-on-North-Sea.htm>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, April 11, 2019 2:16 PM
> *To:* Klaij, Christiaan
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Error with KSPSetUp and MatNest
>
> On Thu, Apr 11, 2019 at 7:51 AM Klaij, Christiaan via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Matt,
>>
>> As a happy MATNEST user, I got a bit worried when you wrote "we
>> should remove MatCreateNest() completely".
>
>
> This would not remove any of the Nest functionality, just the direct
> interface to it, which is the problem.
>
>
>> This happened last
>> time I tried to use AIJ instead:
>>
>>
>> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-August/029973.html
>>
>> Has this problem been fixed in the meantime?
>>
>
> This looks like a problem with the Nest implementation, not AIJ? Maybe I
> am misunderstanding it.
>
>    Thanks,
>
>      Matt
>
>
>> Chris
>>
>> > I think we should remove MatCreateNest() completely. We should have 1
>> > mechanism for getting submatrices for creation, not 2, so we retain
>> > MatCreateLocalRef(). Then if -mat_type nest was specified, it gives you
>> > back an actually submatrix, otherwise it gives a view. This would do
>> > everything that we currently do without this horrible MatNest interface
>> > bubbling to the top.
>> >
>> >    Matt
>> >
>>
>>
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl |
>> http://www.marin.nl
>>
>> MARIN news:
>> http://www.marin.nl/web/News/News-items/42nd-FPSO-JIP-Week-April-812-Singapore.htm
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
> *Help us improve the spam filter. If this message contains SPAM, click
> here
> <https://www.mailcontrol.com/sr/VaxGuPX0yKPGX2PQPOmvUqEZpHzWxmiI7TcWSqBM5JDxEDemOSATW7gNYv05LkIR8iuercSaQ0xkxH4nYzTQYQ==>
> to report. Thank you, MARIN Support Group*
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/b473f4ae/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagea896fb.PNG
Type: image/png
Size: 293 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/b473f4ae/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image0539c8.PNG
Type: image/png
Size: 331 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/b473f4ae/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagefee82c.PNG
Type: image/png
Size: 333 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/b473f4ae/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image188b81.PNG
Type: image/png
Size: 253 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/b473f4ae/attachment-0007.png>

From sajidsyed2021 at u.northwestern.edu  Thu Apr 11 10:39:45 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 11 Apr 2019 10:39:45 -0500
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <604E1BB8-F0D5-4A6E-A62B-4AFC6965E2C4@mcs.anl.gov>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
	<477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>
	<CAOGsD9iQtN8jZOjsYF18jebBUCQivHEs8dE7s+mnZHoRW+-Txg@mail.gmail.com>
	<604E1BB8-F0D5-4A6E-A62B-4AFC6965E2C4@mcs.anl.gov>
Message-ID: <CAOGsD9gSmgpHoZqguExr9PQ93s+495EQ7UpL=MCSvP84y21dXA@mail.gmail.com>

One last question I have is : Does PETSc automatically chose a good chunk
size for the size of the vector it has and use it to write the dataset ? Or
is this something I shouldn't really worry about (not that it affects me
now but it would be good to not have a slow read from a python script for
post-processing).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/90f312ce/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 11 11:31:13 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 11 Apr 2019 16:31:13 +0000
Subject: [petsc-users] Argument out of range error only in certain mpi
 sizes
In-Reply-To: <CAOGsD9gSmgpHoZqguExr9PQ93s+495EQ7UpL=MCSvP84y21dXA@mail.gmail.com>
References: <CAOGsD9h-+o=K1T_c4UuhNa29hrGjenQ5PD+E4GTZ=cm7SLA5FQ@mail.gmail.com>
	<477C39B9-1C4C-42AB-9F80-F34B3B81C056@anl.gov>
	<CAOGsD9iQtN8jZOjsYF18jebBUCQivHEs8dE7s+mnZHoRW+-Txg@mail.gmail.com>
	<604E1BB8-F0D5-4A6E-A62B-4AFC6965E2C4@mcs.anl.gov>
	<CAOGsD9gSmgpHoZqguExr9PQ93s+495EQ7UpL=MCSvP84y21dXA@mail.gmail.com>
Message-ID: <D9670362-6E2F-451A-8039-6466AAC0747B@mcs.anl.gov>


  There is a rather detailed routine for computing the HDF5 chunk size that PETSc uses for its parallel IO in VecGetHDF5ChunkSize() which is called from VecView_MPI_HDF5_DA(). (Note the this attempts to optimize it for the parallel IO that PETSc is doing). I don't understand HDF5 chunk sizes so don't know the relationship between a chunk size used for writing to a HDF5 file and reading from it.

    Barry


> On Apr 11, 2019, at 10:39 AM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> One last question I have is : Does PETSc automatically chose a good chunk size for the size of the vector it has and use it to write the dataset ? Or is this something I shouldn't really worry about (not that it affects me now but it would be good to not have a slow read from a python script for post-processing).
> 


From yyang85 at stanford.edu  Thu Apr 11 17:44:44 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 11 Apr 2019 22:44:44 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
Message-ID: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I'd like to check if it's ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?

Also, I know I've asked this before about linking to MPICH when I call mpirun, instead of using my computer's default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn't work.)

The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don't really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):

==830== Invalid read of size 8
==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
==830==    by 0x62BA574: PCSetUp (precon.c:932)
==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)

==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd

==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
==830==    by 0x62BA574: PCSetUp (precon.c:932)
==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)

Thank you!
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/4e74462b/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Thu Apr 11 18:49:31 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 11 Apr 2019 18:49:31 -0500
Subject: [petsc-users] How to build FFTW3 interface?
Message-ID: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>

Hi PETSc Developers,

To run an example that involves the petsc-fftw interface, I loaded both
petsc and fftw modules (linked of course to the same mpi) but the compiler
complains of having no knowledge of functions like MatCreateVecsFFTW which
happens to be defined at (in the source repo)
petsc/src/mat/impls/fft/fftw.c. I don't see a corresponding definition in
the install folder (I may be wrong, but i just did a simple grep to find
the definition of the function I'm looking for and didn't find it while it
was present in the header and example files).

>From previous threads on this list-serv I see that the developers asked
users to use --download-fftw at configure time, but for users that already
have an fftw installed, is there an option to ask petsc to build the
interfaces as well (I didn't see any such option listed either here:
https://www.mcs.anl.gov/petsc/documentation/installation.html or a variant
in spack) ?

Also, could the fftw version to download be bumped to 3.3.8 (here :
petsc/config/BuildSystem/config/packages/fftw.py) since 3.3.7 gives
erroneous results with gcc-8.

Bug in fftw-3.3.7+gcc-8 :
https://github.com/FFTW/fftw3/commit/19eeeca592f63413698f23dd02b9961f22581803


Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/9e5cbd23/attachment.html>

From mfadams at lbl.gov  Thu Apr 11 20:44:25 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 11 Apr 2019 21:44:25 -0400
Subject: [petsc-users] How to build FFTW3 interface?
In-Reply-To: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
References: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
Message-ID: <CADOhEh4-yRVGxL=Q=irEhzdOHzVReOq_JepLzRVsVR+6uDsw0A@mail.gmail.com>

On Thu, Apr 11, 2019 at 7:51 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi PETSc Developers,
>
> To run an example that involves the petsc-fftw interface, I loaded both
> petsc and fftw modules (linked of course to the same mpi) but the compiler
> complains of having no knowledge of functions like MatCreateVecsFFTW which
> happens to be defined at (in the source repo)
> petsc/src/mat/impls/fft/fftw.c. I don't see a corresponding definition in
> the install folder (I may be wrong, but i just did a simple grep to find
> the definition of the function I'm looking for and didn't find it while it
> was present in the header and example files).
>
> From previous threads on this list-serv I see that the developers asked
> users to use --download-fftw at configure time, but for users that already
> have an fftw installed, is there an option to ask petsc to build the
> interfaces as well (I didn't see any such option listed either here:
> https://www.mcs.anl.gov/petsc/documentation/installation.html or a
> variant in spack) ?
>

Your PETSc needs to be configured with something like --with-fftw-dir=/....
or --download-fftw


>
> Also, could the fftw version to download be bumped to 3.3.8 (here :
> petsc/config/BuildSystem/config/packages/fftw.py) since 3.3.7 gives
> erroneous results with gcc-8.
>
> Bug in fftw-3.3.7+gcc-8 :
> https://github.com/FFTW/fftw3/commit/19eeeca592f63413698f23dd02b9961f22581803
>
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190411/452be759/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 11 20:44:54 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 12 Apr 2019 01:44:54 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>



> On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello team,
>  
> I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?

   Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example

  KSPSetFromOptions(ksp);
  KSPSetOperators(ksp,A,A);
  KSPSolve(ksp,b,x); 
  KSPReset(ksp);
  KSPSetOperators(ksp,B,B);
  KSPSolve(ksp,newb,newx);

>  
> Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)

    Are you asking how you can avoid something like

      /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?

   You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file

      export PATH=/home/me/petsc/arch-myarch/bin:$PATH
      mpiexec -n 2 ./mycode

>  
> The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
>  
> ==830== Invalid read of size 8
> ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>  
> ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
>  
> ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)

   This is curious. The line in the MUMPS code where valgrind detects a problem is 

            K = 1_8
            THEMIN = ZERO
            DO
               IF(THEMIN .NE. ZERO) EXIT
               THEMIN = abs(id%A(K))                               <<<<<<< this line
               K = K+1_8

   So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states 
> 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
problem size is super small so its not like it might have run out of memory)

    What happens if you allow the valgrind to continue? Do you get more valgrind errors?

    What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to 
produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
reasonable. 

   Barry




>  
> Thank you!
> Yuyun


From balay at mcs.anl.gov  Thu Apr 11 21:02:01 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 12 Apr 2019 02:02:01 +0000
Subject: [petsc-users] How to build FFTW3 interface?
In-Reply-To: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
References: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904112054170.8250@sb>

On Thu, 11 Apr 2019, Sajid Ali via petsc-users wrote:

> Hi PETSc Developers,
> 
> To run an example that involves the petsc-fftw interface, I loaded both
> petsc and fftw modules (linked of course to the same mpi) but the compiler
> complains of having no knowledge of functions like MatCreateVecsFFTW which
> happens to be defined at (in the source repo)
> petsc/src/mat/impls/fft/fftw.c. 

Sure. Unless petsc is built with fftw enabled - this interface is not enabled.

> I don't see a corresponding definition in
> the install folder (I may be wrong, but i just did a simple grep to find
> the definition of the function I'm looking for and didn't find it while it
> was present in the header and example files).
> 
> >From previous threads on this list-serv I see that the developers asked
> users to use --download-fftw at configure time, but for users that already
> have an fftw installed, is there an option to ask petsc to build the
> interfaces as well (I didn't see any such option listed either here:
> https://www.mcs.anl.gov/petsc/documentation/installation.html or a variant
> in spack) ?

If you are build petsc from source the configure option is --with-fftw-dir=/location

or let petsc install it with --download-fftw.

> 
> Also, could the fftw version to download be bumped to 3.3.8 (here :
> petsc/config/BuildSystem/config/packages/fftw.py) since 3.3.7 gives
> erroneous results with gcc-8.
> 
> Bug in fftw-3.3.7+gcc-8 :
> https://github.com/FFTW/fftw3/commit/19eeeca592f63413698f23dd02b9961f22581803

Wrt petsc configure - you can try:
./configure --download-fftw=http://www.fftw.org/fftw-3.3.8.tar.gz

Wrt spack - you can try the branch 'balay/petsc-fftw' and see if it works for you

spack install petsc+fftw

Satish

From yyang85 at stanford.edu  Thu Apr 11 23:17:14 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 12 Apr 2019 04:17:14 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
Message-ID: <DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>

Thanks Barry for the detailed answers!

Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.

Thanks again for the timely help!
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Thursday, April 11, 2019 6:44:54 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH



> On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Hello team,
>
> I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?

   Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example

  KSPSetFromOptions(ksp);
  KSPSetOperators(ksp,A,A);
  KSPSolve(ksp,b,x);
  KSPReset(ksp);
  KSPSetOperators(ksp,B,B);
  KSPSolve(ksp,newb,newx);

>
> Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)

    Are you asking how you can avoid something like

      /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?

   You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file

      export PATH=/home/me/petsc/arch-myarch/bin:$PATH
      mpiexec -n 2 ./mycode

>
> The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
>
> ==830== Invalid read of size 8
> ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>
> ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
>
> ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)

   This is curious. The line in the MUMPS code where valgrind detects a problem is

            K = 1_8
            THEMIN = ZERO
            DO
               IF(THEMIN .NE. ZERO) EXIT
               THEMIN = abs(id%A(K))                               <<<<<<< this line
               K = K+1_8

   So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states
> 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
problem size is super small so its not like it might have run out of memory)

    What happens if you allow the valgrind to continue? Do you get more valgrind errors?

    What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to
produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
reasonable.

   Barry




>
> Thank you!
> Yuyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190412/7c0c1289/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 11 23:21:11 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 12 Apr 2019 04:21:11 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>


   Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.

   Barry



> On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> Thanks Barry for the detailed answers!
> 
> Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
> 
> Thanks again for the timely help!
> Yuyun
> 
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 6:44:54 PM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
>  
> 
> 
> > On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello team,
> >  
> > I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
> 
>    Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example
> 
>   KSPSetFromOptions(ksp);
>   KSPSetOperators(ksp,A,A);
>   KSPSolve(ksp,b,x); 
>   KSPReset(ksp);
>   KSPSetOperators(ksp,B,B);
>   KSPSolve(ksp,newb,newx);
> 
> >  
> > Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)
> 
>     Are you asking how you can avoid something like
> 
>       /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
> 
>    You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file
> 
>       export PATH=/home/me/petsc/arch-myarch/bin:$PATH
>       mpiexec -n 2 ./mycode
> 
> >  
> > The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
> >  
> > ==830== Invalid read of size 8
> > ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> > ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> > ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> > ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> > ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> > ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> >  
> > ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
> >  
> > ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> > ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> > ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> > ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
> 
>    This is curious. The line in the MUMPS code where valgrind detects a problem is 
> 
>             K = 1_8
>             THEMIN = ZERO
>             DO
>                IF(THEMIN .NE. ZERO) EXIT
>                THEMIN = abs(id%A(K))                               <<<<<<< this line
>                K = K+1_8
> 
>    So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states 
> > 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
> the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
> problem size is super small so its not like it might have run out of memory)
> 
>     What happens if you allow the valgrind to continue? Do you get more valgrind errors?
> 
>     What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to 
> produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
> reasonable. 
> 
>    Barry
> 
> 
> 
> 
> >  
> > Thank you!
> > Yuyun


From yyang85 at stanford.edu  Fri Apr 12 00:01:30 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 12 Apr 2019 05:01:30 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
Message-ID: <DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>

I think this problem arose because I did not reset the ksp for solving a different problem! It's not giving me an error anymore now that I added the reset, so it's all good :)

Thanks,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Thursday, April 11, 2019 9:21:11 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH


   Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.

   Barry



> On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>
> Thanks Barry for the detailed answers!
>
> Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
>
> Thanks again for the timely help!
> Yuyun
>
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 6:44:54 PM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
>
>
>
> > On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> >
> > Hello team,
> >
> > I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
>
>    Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example
>
>   KSPSetFromOptions(ksp);
>   KSPSetOperators(ksp,A,A);
>   KSPSolve(ksp,b,x);
>   KSPReset(ksp);
>   KSPSetOperators(ksp,B,B);
>   KSPSolve(ksp,newb,newx);
>
> >
> > Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)
>
>     Are you asking how you can avoid something like
>
>       /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
>
>    You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file
>
>       export PATH=/home/me/petsc/arch-myarch/bin:$PATH
>       mpiexec -n 2 ./mycode
>
> >
> > The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
> >
> > ==830== Invalid read of size 8
> > ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> > ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> > ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> > ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> > ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> > ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> >
> > ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
> >
> > ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> > ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> > ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> > ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
>
>    This is curious. The line in the MUMPS code where valgrind detects a problem is
>
>             K = 1_8
>             THEMIN = ZERO
>             DO
>                IF(THEMIN .NE. ZERO) EXIT
>                THEMIN = abs(id%A(K))                               <<<<<<< this line
>                K = K+1_8
>
>    So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states
> > 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
> the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
> problem size is super small so its not like it might have run out of memory)
>
>     What happens if you allow the valgrind to continue? Do you get more valgrind errors?
>
>     What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to
> produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
> reasonable.
>
>    Barry
>
>
>
>
> >
> > Thank you!
> > Yuyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190412/b52bd10a/attachment.html>

From bsmith at mcs.anl.gov  Fri Apr 12 00:18:24 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 12 Apr 2019 05:18:24 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB524210D5CCB02CDB048951FDD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB524210D5CCB02CDB048951FDD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <7FD6886B-30E5-4F96-8ED0-E2645EFC29E9@mcs.anl.gov>


   This means the mpiexec in your path (use which mpiexec for find exactly which one it is) is not associated with the same MPI that PETSc was ./configured with. Make sure that PETSC_DIR and PETSC_ARCH are defined in your .bashrc file before the PATH is defined and make sure each time you edit the .bashrc file you run source ~/.bashrc

   Barry


> On Apr 12, 2019, at 12:13 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I do have a follow-up question regarding MPICH. I?ve set the PATH in my bashrc file according to the suggestion. When I call mpiexec -n 2 ./main, however, it does not seem like the two processors are splitting up the work. Rather, they seem to still be solving the same problem individually with different speeds. Please see a snapshot of some of the results here (it?s printing out the number of time steps taken, time and delta T):
>  
> 40: t = 2.757997927832237e+01 s, dt = 3.02226e+00
> 43: t = 3.930627798681907e+01 s, dt = 4.41084e+00
> 41: t = 3.100752205217186e+01 s, dt = 3.42754e+00
> 44: t = 4.431113388879382e+01 s, dt = 5.00486e+00
> 42: t = 3.489543343038465e+01 s, dt = 3.88791e+00
> 45: t = 4.999071244027611e+01 s, dt = 5.67958e+00
> 43: t = 3.930627798681907e+01 s, dt = 4.41084e+00
> 46: t = 5.643670313814994e+01 s, dt = 6.44599e+00
> 44: t = 4.431113388879382e+01 s, dt = 5.00486e+00
> 47: t = 6.375326587559283e+01 s, dt = 7.31656e+00
> 45: t = 4.999071244027611e+01 s, dt = 5.67958e+00
> 48: t = 7.205870073386113e+01 s, dt = 8.30543e+00
> 46: t = 5.643670313814994e+01 s, dt = 6.44599e+00
> 49: t = 8.148735633607836e+01 s, dt = 9.42866e+00
> 47: t = 6.375326587559283e+01 s, dt = 7.31656e+00
> 50: t = 9.219190788945764e+01 s, dt = 1.07046e+01
> 48: t = 7.205870073386113e+01 s, dt = 8.30543e+00
>  
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Yuyun Yang via petsc-users
> Sent: Thursday, April 11, 2019 10:02 PM
> To: Smith, Barry F. <bsmith at mcs.anl.gov>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
>  
> I think this problem arose because I did not reset the ksp for solving a different problem! It's not giving me an error anymore now that I added the reset, so it's all good :)
>  
> Thanks,
> Yuyun
>  
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 9:21:11 PM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
>  
> 
>    Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
> 
>    Barry
> 
> 
> 
> > On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> > 
> > Thanks Barry for the detailed answers!
> > 
> > Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
> > 
> > Thanks again for the timely help!
> > Yuyun
> > 
> > Get Outlook for iOS
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Thursday, April 11, 2019 6:44:54 PM
> > To: Yuyun Yang
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
> >  
> > 
> > 
> > > On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > Hello team,
> > >  
> > > I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
> > 
> >    Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example
> > 
> >   KSPSetFromOptions(ksp);
> >   KSPSetOperators(ksp,A,A);
> >   KSPSolve(ksp,b,x); 
> >   KSPReset(ksp);
> >   KSPSetOperators(ksp,B,B);
> >   KSPSolve(ksp,newb,newx);
> > 
> > >  
> > > Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)
> > 
> >     Are you asking how you can avoid something like
> > 
> >       /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
> > 
> >    You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file
> > 
> >       export PATH=/home/me/petsc/arch-myarch/bin:$PATH
> >       mpiexec -n 2 ./mycode
> > 
> > >  
> > > The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
> > >  
> > > ==830== Invalid read of size 8
> > > ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> > > ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> > > ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> > > ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> > > ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> > > ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > >  
> > > ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
> > >  
> > > ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > > ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> > > ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> > > ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > > ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> > > ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
> > 
> >    This is curious. The line in the MUMPS code where valgrind detects a problem is 
> > 
> >             K = 1_8
> >             THEMIN = ZERO
> >             DO
> >                IF(THEMIN .NE. ZERO) EXIT
> >                THEMIN = abs(id%A(K))                               <<<<<<< this line
> >                K = K+1_8
> > 
> >    So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states 
> > > 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
> > the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
> > problem size is super small so its not like it might have run out of memory)
> > 
> >     What happens if you allow the valgrind to continue? Do you get more valgrind errors?
> > 
> >     What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to 
> > produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
> > reasonable. 
> > 
> >    Barry
> > 
> > 
> > 
> > 
> > >  
> > > Thank you!
> > > Yuyun
> 


From bsmith at mcs.anl.gov  Fri Apr 12 00:21:04 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 12 Apr 2019 05:21:04 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>


  Ok, still a little odd. PCSetOperators() which is called internally by KSPSetOperators() checks if the matrix has changed size and generates an error. Similar if you set a different matrix from before it resets the computation of the preconditioner. So, in theory, your situation should never occur.

   Barry


> On Apr 12, 2019, at 12:01 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I think this problem arose because I did not reset the ksp for solving a different problem! It's not giving me an error anymore now that I added the reset, so it's all good :)
> 
> Thanks,
> Yuyun
> 
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 9:21:11 PM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
>  
> 
>    Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
> 
>    Barry
> 
> 
> 
> > On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> > 
> > Thanks Barry for the detailed answers!
> > 
> > Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
> > 
> > Thanks again for the timely help!
> > Yuyun
> > 
> > Get Outlook for iOS
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Thursday, April 11, 2019 6:44:54 PM
> > To: Yuyun Yang
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
> >  
> > 
> > 
> > > On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > Hello team,
> > >  
> > > I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
> > 
> >    Do you mean call KSPSetOperators() with one matrix and then later call it with a different matrix? This is ok if the two matrices are the same size and have the same parallel layout. But if the matrices are different size, have different parallel layout then you need to destroy the KSP and create a new one or call KSPReset() in between for example
> > 
> >   KSPSetFromOptions(ksp);
> >   KSPSetOperators(ksp,A,A);
> >   KSPSolve(ksp,b,x); 
> >   KSPReset(ksp);
> >   KSPSetOperators(ksp,B,B);
> >   KSPSolve(ksp,newb,newx);
> > 
> > >  
> > > Also, I know I?ve asked this before about linking to MPICH when I call mpirun, instead of using my computer?s default MPI, but I want to check again. The same problem was solved on my cluster by using a different CLINKER (called mpiicc) in the Makefile and a different intel compiler, which will link my compiled code with MPICH. Is there a similar thing I can do on my own computer, instead of having to use a very long path to locate the MPICH I configured with PETSc, and then calling the executable? (I tried making CLINKER = mpiicc on my own computer but that didn?t work.)
> > 
> >     Are you asking how you can avoid something like
> > 
> >       /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
> > 
> >    You can add /home/me/petsc/arch-myarch/bin to the beginning of your PATH, for example with bash put the following into your ~/.bashrc file
> > 
> >       export PATH=/home/me/petsc/arch-myarch/bin:$PATH
> >       mpiexec -n 2 ./mycode
> > 
> > >  
> > > The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
> > >  
> > > ==830== Invalid read of size 8
> > > ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> > > ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> > > ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> > > ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> > > ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> > > ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > >  
> > > ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 alloc'd
> > >  
> > > ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > > ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> > > ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> > > ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > > ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> > > ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
> > 
> >    This is curious. The line in the MUMPS code where valgrind detects a problem is 
> > 
> >             K = 1_8
> >             THEMIN = ZERO
> >             DO
> >                IF(THEMIN .NE. ZERO) EXIT
> >                THEMIN = abs(id%A(K))                               <<<<<<< this line
> >                K = K+1_8
> > 
> >    So it has a problem accessing id%A(1)  the very first entry in numerical values of the sparse matrix. Meanwhile it states 
> > > 0 bytes after a block of size 7,872 alloc'd MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where PETSc allocates
> > the values passed to MUMPS. So it almost as if MUMPS never allocated any space for id%A(), I can't imagine why that would ever happen (the
> > problem size is super small so its not like it might have run out of memory)
> > 
> >     What happens if you allow the valgrind to continue? Do you get more valgrind errors?
> > 
> >     What happens if run without valgrind? Does it crash at this point in the code? At some later point? Does it run to completion and seem to 
> > produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look
> > reasonable. 
> > 
> >    Barry
> > 
> > 
> > 
> > 
> > >  
> > > Thank you!
> > > Yuyun


From myriam.peyrounette at idris.fr  Fri Apr 12 06:57:05 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Fri, 12 Apr 2019 13:57:05 +0200
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <87zhox5gxi.fsf@jedbrown.org>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
Message-ID: <93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>

Hi all,

I used the wrong script, that's why it diverged... Sorry about that.?
I tried again with the right script applied on a tiny problem (~200
elements). I can see a small difference in memory usage (gain ~ 1mB).
when adding the -mat_freeintermediatestructures option. I still have to
execute larger cases to plot the scaling. The supercomputer I am used to
run my jobs on is really busy at the moment so it takes a while. I hope
I'll send you the results on Monday.

Thanks everyone,

Myriam


Le 04/11/19 ? 06:01, Jed Brown a ?crit?:
> "Zhang, Hong" <hzhang at mcs.anl.gov> writes:
>
>> Jed:
>>>> Myriam,
>>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
>>> And yet if turning it on causes divergence, that would imply a bug.
>>> Hong, are you able to reproduce the experiment to see the memory
>>> scaling?
>> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.
> I have now compiled and run her code locally.
>
> Myriam, thanks for your last mail adding configuration and removing the
> MemManager.h dependency.  I ran with and without
> -mat_freeintermediatedatastructures and don't see a difference in
> convergence.  What commands did you run to observe that difference?

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190412/88ec5542/attachment.p7s>

From sajidsyed2021 at u.northwestern.edu  Fri Apr 12 09:51:01 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 12 Apr 2019 09:51:01 -0500
Subject: [petsc-users] How to build FFTW3 interface?
In-Reply-To: <alpine.LFD.2.21.1904112054170.8250@sb>
References: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
	<alpine.LFD.2.21.1904112054170.8250@sb>
Message-ID: <CAOGsD9iCsEYSZQJPWjA+LWfZQGGxsgZsg3QFpzLpb3zkreB+5g@mail.gmail.com>

Hi Balay,

Confirming that the spack variant works. Thanks for adding it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190412/b3560c8a/attachment.html>

From balay at mcs.anl.gov  Fri Apr 12 09:55:09 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 12 Apr 2019 14:55:09 +0000
Subject: [petsc-users] How to build FFTW3 interface?
In-Reply-To: <CAOGsD9iCsEYSZQJPWjA+LWfZQGGxsgZsg3QFpzLpb3zkreB+5g@mail.gmail.com>
References: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
	<alpine.LFD.2.21.1904112054170.8250@sb>
	<CAOGsD9iCsEYSZQJPWjA+LWfZQGGxsgZsg3QFpzLpb3zkreB+5g@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904120954500.8399@sb>

Great! The change is now in spack 'develop' branch.

Satish

On Fri, 12 Apr 2019, Sajid Ali via petsc-users wrote:

> Hi Balay,
> 
> Confirming that the spack variant works. Thanks for adding it.
> 


From balay at mcs.anl.gov  Fri Apr 12 10:00:50 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 12 Apr 2019 15:00:50 +0000
Subject: [petsc-users] petsc-3.11.1.tar.gz now available
Message-ID: <alpine.LFD.2.21.1904120956150.8399@sb>

Dear PETSc users,

The patch release petsc-3.11.1 is now available for download,
with change list at 'PETSc-3.11 Changelog'

http://www.mcs.anl.gov/petsc/download/index.html

Satish

From hzhang at mcs.anl.gov  Fri Apr 12 11:18:50 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Fri, 12 Apr 2019 16:18:50 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<d5aef95f-4022-1c99-018b-71c2d3d38f43@idris.fr>
	<CAJ98EDr+0cna4ndHkCUULV3DMU0APfgTh-N9Dr8AwVAFwh-+tQ@mail.gmail.com>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
Message-ID: <CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>

Myriam :
Thanks for your effort. It will help us improve PETSc.
Hong

Hi all,

I used the wrong script, that's why it diverged... Sorry about that.
I tried again with the right script applied on a tiny problem (~200
elements). I can see a small difference in memory usage (gain ~ 1mB).
when adding the -mat_freeintermediatestructures option. I still have to
execute larger cases to plot the scaling. The supercomputer I am used to
run my jobs on is really busy at the moment so it takes a while. I hope
I'll send you the results on Monday.

Thanks everyone,

Myriam


Le 04/11/19 ? 06:01, Jed Brown a ?crit :
> "Zhang, Hong" <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> writes:
>
>> Jed:
>>>> Myriam,
>>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
>>> And yet if turning it on causes divergence, that would imply a bug.
>>> Hong, are you able to reproduce the experiment to see the memory
>>> scaling?
>> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.
> I have now compiled and run her code locally.
>
> Myriam, thanks for your last mail adding configuration and removing the
> MemManager.h dependency.  I ran with and without
> -mat_freeintermediatedatastructures and don't see a difference in
> convergence.  What commands did you run to observe that difference?

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190412/5d524ad4/attachment.html>

From fuentesdt at gmail.com  Sat Apr 13 05:07:49 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Sat, 13 Apr 2019 05:07:49 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
	<CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
Message-ID: <CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>

this refinement seems to work pretty well for what I was looking for.
still testing... but tetgen seems to load these  background mesh files at
the higher resolution to guide the refinement?

$ tetgen -Vrmqk coarse.1.node
Opening coarse.1.node.
Opening coarse.1.ele.
Opening coarse.1.face.
Opening coarse.1.b.node.
Opening coarse.1.b.ele.
Opening coarse.1.b.mtr.
.
.

do you think a reasonable approach would be for me work with two meshes in
petsc? a coarse and fine mesh. and load a uniformly refined fine dmplex
mesh as the background mesh?

https://bitbucket.org/petsc/ctetgen/src/8ca66eb7de95b82f7969984bab6aacada6626b1d/ctetgen.c?at=master&fileviewer=file-view-default#ctetgen.c-22245

[image: Screen Shot 2019-04-13 at 4.50.06 AM.png]

On Tue, Jan 8, 2019 at 11:29 AM David Fuentes <fuentesdt at gmail.com> wrote:

> sounds great! i've been working out of this guy -
> https://github.com/fuentesdt/thermoembo
>
> and loading vtk data for the segmentation:
> https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50
>
> I can clean this up and separate into relevant directories ? or start an
> new repo? which ever you prefer.
>
>
>
> On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com> wrote:
>>
>>> ha! thanks for you time on this Matt. I'm trying to generate a mesh from
>>> image segmentation data.
>>> I would like to use an image segmentation to guide the refinement.
>>> Figure 25 of this paper -
>>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>>
>>
>> Very cool. We can do that. Lets make an example and iterate. Want to make
>> a repo you control? I will
>> stick in a PETSc example that refines meshes (maybe Plex ex19), and we
>> can experiment with both p4est
>> and Pragmatic (installation of those is hardest part :) This sounds great.
>>
>>    Matt
>>
>>
>>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>>> wrote:
>>>>
>>>>> thanks Matt,
>>>>>
>>>>> I posted a slightly modified example
>>>>>
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>>
>>>>> and changes from the orginal
>>>>>
>>>>>
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>>
>>>>>
>>>>> This is what I'm seeing
>>>>>
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>>
>>>>>
>>>>> I'm refining based on this element centroid:
>>>>>
>>>>>
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>>
>>>>> unrefined -
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>>
>>>>> refined -
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>>
>>>>>
>>>>> Maybe I don't understand the refinement algorithm, but the refinement
>>>>> is a little offset from what I was expecting.
>>>>>
>>>>
>>>> Okay, I have that working. I will make a branch and put this in it. For
>>>> right now, I attach ex45.c (but it will not run since I put in stuff to
>>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>>> can see it is refining cell 7, not 8. Its just that refinement
>>>> propagates a long, long way.
>>>>
>>>> I have pretty much abandoned this type of refinement since it will not
>>>> work in parallel, and the mesh generator
>>>> interface is poor. I am transitioning everything to
>>>>
>>>>   - uniform refinement (ha!)
>>>>   - cell marking (p4est likes this)
>>>>   - metric tensor (Pragmatic likes this)
>>>>
>>>> I have some routines to convert marking <--> metric but they are not
>>>> perfect I think. I can try and help do what you want with
>>>> AMR if I have a better idea what it is.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>>
>>>>> Also, can you pass an application context to the user refinement
>>>>> function ?
>>>>>
>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Starting with TS example 45
>>>>>>>
>>>>>>> $ pwd
>>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>>
>>>>>>> petsc configured with: ./config/configure.py --with-shared-libraries
>>>>>>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>>>>>>> --download-ctetgen  --download-triangle --with-debugging=yes
>>>>>>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>>>>>>> --download-pnetcdf
>>>>>>>
>>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>>> DMPlexSetRefinementFunction.
>>>>>>> It generally seems to be working, except that the refined element is
>>>>>>> slightly offset from what I was expecting.
>>>>>>> Based on my application specific criteria, element id number 7 is
>>>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>>>> pic.
>>>>>>>
>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>>
>>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>>> transfering to tetgen data structures somehow ?
>>>>>>>
>>>>>>
>>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>>> modified example and I will walk through it with the
>>>>>> debugger?
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> (gdb) bt
>>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>>> dmf=0x7fffffffb938) at
>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/9c895c8f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-04-13 at 4.50.06 AM.png
Type: image/png
Size: 84446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/9c895c8f/attachment-0001.png>

From knepley at gmail.com  Sat Apr 13 06:25:08 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 13 Apr 2019 07:25:08 -0400
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
	<CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
	<CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>
Message-ID: <CAMYG4GmV71JNzLY1-UnBY1T+sZyxcrXLiy7e83bhFrPc66ex+g@mail.gmail.com>

On Sat, Apr 13, 2019 at 6:08 AM David Fuentes <fuentesdt at gmail.com> wrote:

> this refinement seems to work pretty well for what I was looking for.
> still testing... but tetgen seems to load these  background mesh files at
> the higher resolution to guide the refinement?
>

Okay, I finally understand what the background mesh does. It gives the mesh
sizing function.


> $ tetgen -Vrmqk coarse.1.node
> Opening coarse.1.node.
> Opening coarse.1.ele.
> Opening coarse.1.face.
> Opening coarse.1.b.node.
> Opening coarse.1.b.ele.
> Opening coarse.1.b.mtr.
>
> do you think a reasonable approach would be for me work with two meshes in
> petsc?
>

Sure, I can turn on that interface for you. I often have a few meshes in
the code.


> a coarse and fine mesh. and load a uniformly refined fine dmplex mesh as
> the background mesh?
>

This is the only thing I dod not understand. If the background mesh is
uniformly refined, can't you just
use a cell volume limit? Maybe I am missing something in the terminology.

   Matt



>
> https://bitbucket.org/petsc/ctetgen/src/8ca66eb7de95b82f7969984bab6aacada6626b1d/ctetgen.c?at=master&fileviewer=file-view-default#ctetgen.c-22245
>
> [image: Screen Shot 2019-04-13 at 4.50.06 AM.png]
>
> On Tue, Jan 8, 2019 at 11:29 AM David Fuentes <fuentesdt at gmail.com> wrote:
>
>> sounds great! i've been working out of this guy -
>> https://github.com/fuentesdt/thermoembo
>>
>> and loading vtk data for the segmentation:
>> https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50
>>
>> I can clean this up and separate into relevant directories ? or start an
>> new repo? which ever you prefer.
>>
>>
>>
>> On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com>
>>> wrote:
>>>
>>>> ha! thanks for you time on this Matt. I'm trying to generate a mesh
>>>> from image segmentation data.
>>>> I would like to use an image segmentation to guide the refinement.
>>>> Figure 25 of this paper -
>>>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>>>
>>>
>>> Very cool. We can do that. Lets make an example and iterate. Want to
>>> make a repo you control? I will
>>> stick in a PETSc example that refines meshes (maybe Plex ex19), and we
>>> can experiment with both p4est
>>> and Pragmatic (installation of those is hardest part :) This sounds
>>> great.
>>>
>>>    Matt
>>>
>>>
>>>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> thanks Matt,
>>>>>>
>>>>>> I posted a slightly modified example
>>>>>>
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>>>
>>>>>> and changes from the orginal
>>>>>>
>>>>>>
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>>>
>>>>>>
>>>>>> This is what I'm seeing
>>>>>>
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>>>
>>>>>>
>>>>>> I'm refining based on this element centroid:
>>>>>>
>>>>>>
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>>>
>>>>>> unrefined -
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>>>
>>>>>> refined -
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>>>
>>>>>>
>>>>>> Maybe I don't understand the refinement algorithm, but the refinement
>>>>>> is a little offset from what I was expecting.
>>>>>>
>>>>>
>>>>> Okay, I have that working. I will make a branch and put this in it.
>>>>> For right now, I attach ex45.c (but it will not run since I put in stuff to
>>>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>>>> can see it is refining cell 7, not 8. Its just that refinement
>>>>> propagates a long, long way.
>>>>>
>>>>> I have pretty much abandoned this type of refinement since it will not
>>>>> work in parallel, and the mesh generator
>>>>> interface is poor. I am transitioning everything to
>>>>>
>>>>>   - uniform refinement (ha!)
>>>>>   - cell marking (p4est likes this)
>>>>>   - metric tensor (Pragmatic likes this)
>>>>>
>>>>> I have some routines to convert marking <--> metric but they are not
>>>>> perfect I think. I can try and help do what you want with
>>>>> AMR if I have a better idea what it is.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>>
>>>>>> Also, can you pass an application context to the user refinement
>>>>>> function ?
>>>>>>
>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Starting with TS example 45
>>>>>>>>
>>>>>>>> $ pwd
>>>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>>>
>>>>>>>> petsc configured with: ./config/configure.py
>>>>>>>> --with-shared-libraries --with-clanguage=c++ --CFLAGS='-g -O0'
>>>>>>>> --CXXFLAGS='-g -O0'  --with-c2html=0 --download-ctetgen
>>>>>>>> --download-triangle --with-debugging=yes --download-netcdf
>>>>>>>> --download-zlib  --download-exodusii --download-hdf5 --download-pnetcdf
>>>>>>>>
>>>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>>>> DMPlexSetRefinementFunction.
>>>>>>>> It generally seems to be working, except that the refined element
>>>>>>>> is slightly offset from what I was expecting.
>>>>>>>> Based on my application specific criteria, element id number 7 is
>>>>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>>>>> pic.
>>>>>>>>
>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>>>
>>>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>>>> transfering to tetgen data structures somehow ?
>>>>>>>>
>>>>>>>
>>>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>>>> modified example and I will walk through it with the
>>>>>>> debugger?
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> (gdb) bt
>>>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>>>> dmf=0x7fffffffb938) at
>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/265bf85c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-04-13 at 4.50.06 AM.png
Type: image/png
Size: 84446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/265bf85c/attachment-0001.png>

From fuentesdt at gmail.com  Sat Apr 13 08:29:41 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Sat, 13 Apr 2019 08:29:41 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAMYG4GmV71JNzLY1-UnBY1T+sZyxcrXLiy7e83bhFrPc66ex+g@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
	<CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
	<CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>
	<CAMYG4GmV71JNzLY1-UnBY1T+sZyxcrXLiy7e83bhFrPc66ex+g@mail.gmail.com>
Message-ID: <CAKnkmZzLr=Lam4AiF6ghrVEwtZvpdTg7Yjzh2-PaKMScqHb_KA@mail.gmail.com>

using this mesh sizing function defined on nodes of the background mesh was
the only way I could get it to work.

http://wias-berlin.de/software/tetgen/1.5/doc/manual/manual006.html#ff_mtr

On Sat, Apr 13, 2019 at 6:25 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Apr 13, 2019 at 6:08 AM David Fuentes <fuentesdt at gmail.com> wrote:
>
>> this refinement seems to work pretty well for what I was looking for.
>> still testing... but tetgen seems to load these  background mesh files at
>> the higher resolution to guide the refinement?
>>
>
> Okay, I finally understand what the background mesh does. It gives the
> mesh sizing function.
>
>
>> $ tetgen -Vrmqk coarse.1.node
>> Opening coarse.1.node.
>> Opening coarse.1.ele.
>> Opening coarse.1.face.
>> Opening coarse.1.b.node.
>> Opening coarse.1.b.ele.
>> Opening coarse.1.b.mtr.
>>
>> do you think a reasonable approach would be for me work with two meshes
>> in petsc?
>>
>
> Sure, I can turn on that interface for you. I often have a few meshes in
> the code.
>
>
>> a coarse and fine mesh. and load a uniformly refined fine dmplex mesh as
>> the background mesh?
>>
>
> This is the only thing I dod not understand. If the background mesh is
> uniformly refined, can't you just
> use a cell volume limit? Maybe I am missing something in the terminology.
>
>    Matt
>
>
>
>>
>> https://bitbucket.org/petsc/ctetgen/src/8ca66eb7de95b82f7969984bab6aacada6626b1d/ctetgen.c?at=master&fileviewer=file-view-default#ctetgen.c-22245
>>
>> [image: Screen Shot 2019-04-13 at 4.50.06 AM.png]
>>
>> On Tue, Jan 8, 2019 at 11:29 AM David Fuentes <fuentesdt at gmail.com>
>> wrote:
>>
>>> sounds great! i've been working out of this guy -
>>> https://github.com/fuentesdt/thermoembo
>>>
>>> and loading vtk data for the segmentation:
>>> https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50
>>>
>>> I can clean this up and separate into relevant directories ? or start an
>>> new repo? which ever you prefer.
>>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com>
>>>> wrote:
>>>>
>>>>> ha! thanks for you time on this Matt. I'm trying to generate a mesh
>>>>> from image segmentation data.
>>>>> I would like to use an image segmentation to guide the refinement.
>>>>> Figure 25 of this paper -
>>>>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>>>>
>>>>
>>>> Very cool. We can do that. Lets make an example and iterate. Want to
>>>> make a repo you control? I will
>>>> stick in a PETSc example that refines meshes (maybe Plex ex19), and we
>>>> can experiment with both p4est
>>>> and Pragmatic (installation of those is hardest part :) This sounds
>>>> great.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> thanks Matt,
>>>>>>>
>>>>>>> I posted a slightly modified example
>>>>>>>
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>>>>
>>>>>>> and changes from the orginal
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>>>>
>>>>>>>
>>>>>>> This is what I'm seeing
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>>>>
>>>>>>>
>>>>>>> I'm refining based on this element centroid:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>>>>
>>>>>>> unrefined -
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>>>>
>>>>>>> refined -
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>>>>
>>>>>>>
>>>>>>> Maybe I don't understand the refinement algorithm, but the
>>>>>>> refinement is a little offset from what I was expecting.
>>>>>>>
>>>>>>
>>>>>> Okay, I have that working. I will make a branch and put this in it.
>>>>>> For right now, I attach ex45.c (but it will not run since I put in stuff to
>>>>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>>>>> can see it is refining cell 7, not 8. Its just that refinement
>>>>>> propagates a long, long way.
>>>>>>
>>>>>> I have pretty much abandoned this type of refinement since it will
>>>>>> not work in parallel, and the mesh generator
>>>>>> interface is poor. I am transitioning everything to
>>>>>>
>>>>>>   - uniform refinement (ha!)
>>>>>>   - cell marking (p4est likes this)
>>>>>>   - metric tensor (Pragmatic likes this)
>>>>>>
>>>>>> I have some routines to convert marking <--> metric but they are not
>>>>>> perfect I think. I can try and help do what you want with
>>>>>> AMR if I have a better idea what it is.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Also, can you pass an application context to the user refinement
>>>>>>> function ?
>>>>>>>
>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Starting with TS example 45
>>>>>>>>>
>>>>>>>>> $ pwd
>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>>>>
>>>>>>>>> petsc configured with: ./config/configure.py
>>>>>>>>> --with-shared-libraries --with-clanguage=c++ --CFLAGS='-g -O0'
>>>>>>>>> --CXXFLAGS='-g -O0'  --with-c2html=0 --download-ctetgen
>>>>>>>>> --download-triangle --with-debugging=yes --download-netcdf
>>>>>>>>> --download-zlib  --download-exodusii --download-hdf5 --download-pnetcdf
>>>>>>>>>
>>>>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>>>>> DMPlexSetRefinementFunction.
>>>>>>>>> It generally seems to be working, except that the refined element
>>>>>>>>> is slightly offset from what I was expecting.
>>>>>>>>> Based on my application specific criteria, element id number 7 is
>>>>>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>>>>>> pic.
>>>>>>>>>
>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>>>>
>>>>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>>>>> transfering to tetgen data structures somehow ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>>>>> modified example and I will walk through it with the
>>>>>>>> debugger?
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (gdb) bt
>>>>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>>>>> dmf=0x7fffffffb938) at
>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/f1f8f131/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-04-13 at 4.50.06 AM.png
Type: image/png
Size: 84446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/f1f8f131/attachment-0001.png>

From knepley at gmail.com  Sat Apr 13 09:26:50 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 13 Apr 2019 10:26:50 -0400
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZzLr=Lam4AiF6ghrVEwtZvpdTg7Yjzh2-PaKMScqHb_KA@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
	<CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
	<CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>
	<CAMYG4GmV71JNzLY1-UnBY1T+sZyxcrXLiy7e83bhFrPc66ex+g@mail.gmail.com>
	<CAKnkmZzLr=Lam4AiF6ghrVEwtZvpdTg7Yjzh2-PaKMScqHb_KA@mail.gmail.com>
Message-ID: <CAMYG4G=6sjQPwDsn6mYH-Vi0JD8YXa_9U8xp=btWYn-7y-PLbQ@mail.gmail.com>

On Sat, Apr 13, 2019 at 9:29 AM David Fuentes <fuentesdt at gmail.com> wrote:

> using this mesh sizing function defined on nodes of the background mesh
> was the only way I could get it to work.
>

Okay, let me understand this. So you could specify a sizing function on the
current mesh. However, TetGen only
does 1 pass, so that gradiation might not be smooth enough. Thus you
specify the sizing function on a refined
background mesh, and get smoother gradiation.

Okay, I can make that work. I have to turn on a bunch of things in CTetGen
that were inactive.

There is an alternative to this. Since you know the sizing function, you
could use Pragmatic instead. The install is
much much harder, but it does work, and works in parallel, and can do
anisotropic refinement as well.

  Thanks,

     Matt



http://wias-berlin.de/software/tetgen/1.5/doc/manual/manual006.html#ff_mtr
>
> On Sat, Apr 13, 2019 at 6:25 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sat, Apr 13, 2019 at 6:08 AM David Fuentes <fuentesdt at gmail.com>
>> wrote:
>>
>>> this refinement seems to work pretty well for what I was looking for.
>>> still testing... but tetgen seems to load these  background mesh files
>>> at the higher resolution to guide the refinement?
>>>
>>
>> Okay, I finally understand what the background mesh does. It gives the
>> mesh sizing function.
>>
>>
>>> $ tetgen -Vrmqk coarse.1.node
>>> Opening coarse.1.node.
>>> Opening coarse.1.ele.
>>> Opening coarse.1.face.
>>> Opening coarse.1.b.node.
>>> Opening coarse.1.b.ele.
>>> Opening coarse.1.b.mtr.
>>>
>>> do you think a reasonable approach would be for me work with two meshes
>>> in petsc?
>>>
>>
>> Sure, I can turn on that interface for you. I often have a few meshes in
>> the code.
>>
>>
>>> a coarse and fine mesh. and load a uniformly refined fine dmplex mesh as
>>> the background mesh?
>>>
>>
>> This is the only thing I dod not understand. If the background mesh is
>> uniformly refined, can't you just
>> use a cell volume limit? Maybe I am missing something in the terminology.
>>
>>    Matt
>>
>>
>>
>>>
>>> https://bitbucket.org/petsc/ctetgen/src/8ca66eb7de95b82f7969984bab6aacada6626b1d/ctetgen.c?at=master&fileviewer=file-view-default#ctetgen.c-22245
>>>
>>> [image: Screen Shot 2019-04-13 at 4.50.06 AM.png]
>>>
>>> On Tue, Jan 8, 2019 at 11:29 AM David Fuentes <fuentesdt at gmail.com>
>>> wrote:
>>>
>>>> sounds great! i've been working out of this guy -
>>>> https://github.com/fuentesdt/thermoembo
>>>>
>>>> and loading vtk data for the segmentation:
>>>> https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50
>>>>
>>>> I can clean this up and separate into relevant directories ? or start
>>>> an new repo? which ever you prefer.
>>>>
>>>>
>>>>
>>>> On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> ha! thanks for you time on this Matt. I'm trying to generate a mesh
>>>>>> from image segmentation data.
>>>>>> I would like to use an image segmentation to guide the refinement.
>>>>>> Figure 25 of this paper -
>>>>>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>>>>>
>>>>>
>>>>> Very cool. We can do that. Lets make an example and iterate. Want to
>>>>> make a repo you control? I will
>>>>> stick in a PETSc example that refines meshes (maybe Plex ex19), and we
>>>>> can experiment with both p4est
>>>>> and Pragmatic (installation of those is hardest part :) This sounds
>>>>> great.
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> thanks Matt,
>>>>>>>>
>>>>>>>> I posted a slightly modified example
>>>>>>>>
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>>>>>
>>>>>>>> and changes from the orginal
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>>>>>
>>>>>>>>
>>>>>>>> This is what I'm seeing
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm refining based on this element centroid:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>>>>>
>>>>>>>> unrefined -
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>>>>>
>>>>>>>> refined -
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>>>>>
>>>>>>>>
>>>>>>>> Maybe I don't understand the refinement algorithm, but the
>>>>>>>> refinement is a little offset from what I was expecting.
>>>>>>>>
>>>>>>>
>>>>>>> Okay, I have that working. I will make a branch and put this in it.
>>>>>>> For right now, I attach ex45.c (but it will not run since I put in stuff to
>>>>>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>>>>>> can see it is refining cell 7, not 8. Its just that refinement
>>>>>>> propagates a long, long way.
>>>>>>>
>>>>>>> I have pretty much abandoned this type of refinement since it will
>>>>>>> not work in parallel, and the mesh generator
>>>>>>> interface is poor. I am transitioning everything to
>>>>>>>
>>>>>>>   - uniform refinement (ha!)
>>>>>>>   - cell marking (p4est likes this)
>>>>>>>   - metric tensor (Pragmatic likes this)
>>>>>>>
>>>>>>> I have some routines to convert marking <--> metric but they are not
>>>>>>> perfect I think. I can try and help do what you want with
>>>>>>> AMR if I have a better idea what it is.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Also, can you pass an application context to the user refinement
>>>>>>>> function ?
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Starting with TS example 45
>>>>>>>>>>
>>>>>>>>>> $ pwd
>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>>>>>
>>>>>>>>>> petsc configured with: ./config/configure.py
>>>>>>>>>> --with-shared-libraries --with-clanguage=c++ --CFLAGS='-g -O0'
>>>>>>>>>> --CXXFLAGS='-g -O0'  --with-c2html=0 --download-ctetgen
>>>>>>>>>> --download-triangle --with-debugging=yes --download-netcdf
>>>>>>>>>> --download-zlib  --download-exodusii --download-hdf5 --download-pnetcdf
>>>>>>>>>>
>>>>>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>>>>>> DMPlexSetRefinementFunction.
>>>>>>>>>> It generally seems to be working, except that the refined element
>>>>>>>>>> is slightly offset from what I was expecting.
>>>>>>>>>> Based on my application specific criteria, element id number 7 is
>>>>>>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>>>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>>>>>>> pic.
>>>>>>>>>>
>>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>>>>>
>>>>>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>>>>>> transfering to tetgen data structures somehow ?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>>>>>> modified example and I will walk through it with the
>>>>>>>>> debugger?
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (gdb) bt
>>>>>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>>>>>> dmf=0x7fffffffb938) at
>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/7ecc0557/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-04-13 at 4.50.06 AM.png
Type: image/png
Size: 84446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/7ecc0557/attachment-0001.png>

From fuentesdt at gmail.com  Sat Apr 13 10:11:03 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Sat, 13 Apr 2019 10:11:03 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAMYG4G=6sjQPwDsn6mYH-Vi0JD8YXa_9U8xp=btWYn-7y-PLbQ@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
	<CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>
	<CAKnkmZxZ_3bpLYzrcCDRPoUFJ2Jyf7QHnZeKZHsN-T20Qq0QMA@mail.gmail.com>
	<CAMYG4GmV71JNzLY1-UnBY1T+sZyxcrXLiy7e83bhFrPc66ex+g@mail.gmail.com>
	<CAKnkmZzLr=Lam4AiF6ghrVEwtZvpdTg7Yjzh2-PaKMScqHb_KA@mail.gmail.com>
	<CAMYG4G=6sjQPwDsn6mYH-Vi0JD8YXa_9U8xp=btWYn-7y-PLbQ@mail.gmail.com>
Message-ID: <CAKnkmZxVoKJSHOhyJjNiZuSnDh-gjvhqhYnfwHtH9fci5dYGAg@mail.gmail.com>

that seems to be what's happening. I wrote a script to load a sizing
function  from an image and two meshes into tetgen. then output an exodus
file to load into dmplex.

https://github.com/fuentesdt/thermoembo/blob/master/tutorials/phasemesh.py#L105



On Sat, Apr 13, 2019 at 9:27 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Apr 13, 2019 at 9:29 AM David Fuentes <fuentesdt at gmail.com> wrote:
>
>> using this mesh sizing function defined on nodes of the background mesh
>> was the only way I could get it to work.
>>
>
> Okay, let me understand this. So you could specify a sizing function on
> the current mesh. However, TetGen only
> does 1 pass, so that gradiation might not be smooth enough. Thus you
> specify the sizing function on a refined
> background mesh, and get smoother gradiation.
>
> Okay, I can make that work. I have to turn on a bunch of things in CTetGen
> that were inactive.
>
> There is an alternative to this. Since you know the sizing function, you
> could use Pragmatic instead. The install is
> much much harder, but it does work, and works in parallel, and can do
> anisotropic refinement as well.
>
>   Thanks,
>
>      Matt
>
>
>
> http://wias-berlin.de/software/tetgen/1.5/doc/manual/manual006.html#ff_mtr
>>
>> On Sat, Apr 13, 2019 at 6:25 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, Apr 13, 2019 at 6:08 AM David Fuentes <fuentesdt at gmail.com>
>>> wrote:
>>>
>>>> this refinement seems to work pretty well for what I was looking for.
>>>> still testing... but tetgen seems to load these  background mesh files
>>>> at the higher resolution to guide the refinement?
>>>>
>>>
>>> Okay, I finally understand what the background mesh does. It gives the
>>> mesh sizing function.
>>>
>>>
>>>> $ tetgen -Vrmqk coarse.1.node
>>>> Opening coarse.1.node.
>>>> Opening coarse.1.ele.
>>>> Opening coarse.1.face.
>>>> Opening coarse.1.b.node.
>>>> Opening coarse.1.b.ele.
>>>> Opening coarse.1.b.mtr.
>>>>
>>>> do you think a reasonable approach would be for me work with two meshes
>>>> in petsc?
>>>>
>>>
>>> Sure, I can turn on that interface for you. I often have a few meshes in
>>> the code.
>>>
>>>
>>>> a coarse and fine mesh. and load a uniformly refined fine dmplex mesh
>>>> as the background mesh?
>>>>
>>>
>>> This is the only thing I dod not understand. If the background mesh is
>>> uniformly refined, can't you just
>>> use a cell volume limit? Maybe I am missing something in the terminology.
>>>
>>>    Matt
>>>
>>>
>>>
>>>>
>>>> https://bitbucket.org/petsc/ctetgen/src/8ca66eb7de95b82f7969984bab6aacada6626b1d/ctetgen.c?at=master&fileviewer=file-view-default#ctetgen.c-22245
>>>>
>>>> [image: Screen Shot 2019-04-13 at 4.50.06 AM.png]
>>>>
>>>> On Tue, Jan 8, 2019 at 11:29 AM David Fuentes <fuentesdt at gmail.com>
>>>> wrote:
>>>>
>>>>> sounds great! i've been working out of this guy -
>>>>> https://github.com/fuentesdt/thermoembo
>>>>>
>>>>> and loading vtk data for the segmentation:
>>>>> https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50
>>>>>
>>>>> I can clean this up and separate into relevant directories ? or start
>>>>> an new repo? which ever you prefer.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> ha! thanks for you time on this Matt. I'm trying to generate a mesh
>>>>>>> from image segmentation data.
>>>>>>> I would like to use an image segmentation to guide the refinement.
>>>>>>> Figure 25 of this paper -
>>>>>>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>>>>>>
>>>>>>
>>>>>> Very cool. We can do that. Lets make an example and iterate. Want to
>>>>>> make a repo you control? I will
>>>>>> stick in a PETSc example that refines meshes (maybe Plex ex19), and
>>>>>> we can experiment with both p4est
>>>>>> and Pragmatic (installation of those is hardest part :) This sounds
>>>>>> great.
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> thanks Matt,
>>>>>>>>>
>>>>>>>>> I posted a slightly modified example
>>>>>>>>>
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>>>>>>
>>>>>>>>> and changes from the orginal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is what I'm seeing
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm refining based on this element centroid:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>>>>>>
>>>>>>>>> unrefined -
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>>>>>>
>>>>>>>>> refined -
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Maybe I don't understand the refinement algorithm, but the
>>>>>>>>> refinement is a little offset from what I was expecting.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Okay, I have that working. I will make a branch and put this in it.
>>>>>>>> For right now, I attach ex45.c (but it will not run since I put in stuff to
>>>>>>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>>>>>>> can see it is refining cell 7, not 8. Its just that refinement
>>>>>>>> propagates a long, long way.
>>>>>>>>
>>>>>>>> I have pretty much abandoned this type of refinement since it will
>>>>>>>> not work in parallel, and the mesh generator
>>>>>>>> interface is poor. I am transitioning everything to
>>>>>>>>
>>>>>>>>   - uniform refinement (ha!)
>>>>>>>>   - cell marking (p4est likes this)
>>>>>>>>   - metric tensor (Pragmatic likes this)
>>>>>>>>
>>>>>>>> I have some routines to convert marking <--> metric but they are
>>>>>>>> not perfect I think. I can try and help do what you want with
>>>>>>>> AMR if I have a better idea what it is.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also, can you pass an application context to the user refinement
>>>>>>>>> function ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Starting with TS example 45
>>>>>>>>>>>
>>>>>>>>>>> $ pwd
>>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>>>>>>
>>>>>>>>>>> petsc configured with: ./config/configure.py
>>>>>>>>>>> --with-shared-libraries --with-clanguage=c++ --CFLAGS='-g -O0'
>>>>>>>>>>> --CXXFLAGS='-g -O0'  --with-c2html=0 --download-ctetgen
>>>>>>>>>>> --download-triangle --with-debugging=yes --download-netcdf
>>>>>>>>>>> --download-zlib  --download-exodusii --download-hdf5 --download-pnetcdf
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>>>>>>> DMPlexSetRefinementFunction.
>>>>>>>>>>> It generally seems to be working, except that the refined
>>>>>>>>>>> element is slightly offset from what I was expecting.
>>>>>>>>>>> Based on my application specific criteria, element id number 7
>>>>>>>>>>> is flagged to be refined by the DMPlexSetRefinementFunction but when I open
>>>>>>>>>>> in paraview, it looks like element id number 8 is being refined. See
>>>>>>>>>>> attached pic.
>>>>>>>>>>>
>>>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>>>>>>
>>>>>>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>>>>>>> transfering to tetgen data structures somehow ?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>>>>>>> modified example and I will walk through it with the
>>>>>>>>>> debugger?
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>>>>>>> dmRefined=0x7fffffffb938) at
>>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>>>>>>> dmf=0x7fffffffb938) at
>>>>>>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/5a9b7269/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-04-13 at 4.50.06 AM.png
Type: image/png
Size: 84446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190413/5a9b7269/attachment-0001.png>

From bsmith at mcs.anl.gov  Sat Apr 13 14:01:46 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 13 Apr 2019 19:01:46 +0000
Subject: [petsc-users] How to build FFTW3 interface?
In-Reply-To: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
References: <CAOGsD9igjDQtUV3=pEAwf0QMtOC6tvHLFu+1hb_q0QdreoK74w@mail.gmail.com>
Message-ID: <BB8C1CDA-D29A-42D7-BB28-BE071C17D605@anl.gov>

https://bitbucket.org/petsc/petsc/pull-requests/1539/update-to-the-lastest-fftw-release/diff


> On Apr 11, 2019, at 6:49 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi PETSc Developers, 
> 
> To run an example that involves the petsc-fftw interface, I loaded both petsc and fftw modules (linked of course to the same mpi) but the compiler complains of having no knowledge of functions like MatCreateVecsFFTW which happens to be defined at (in the source repo) petsc/src/mat/impls/fft/fftw.c. I don't see a corresponding definition in the install folder (I may be wrong, but i just did a simple grep to find the definition of the function I'm looking for and didn't find it while it was present in the header and example files). 
> 
> From previous threads on this list-serv I see that the developers asked users to use --download-fftw at configure time, but for users that already have an fftw installed, is there an option to ask petsc to build the interfaces as well (I didn't see any such option listed either here: https://www.mcs.anl.gov/petsc/documentation/installation.html or a variant in spack) ?
> 
> Also, could the fftw version to download be bumped to 3.3.8 (here : petsc/config/BuildSystem/config/packages/fftw.py) since 3.3.7 gives erroneous results with gcc-8. 
> 
> Bug in fftw-3.3.7+gcc-8 : https://github.com/FFTW/fftw3/commit/19eeeca592f63413698f23dd02b9961f22581803
> 
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University


From yyang85 at stanford.edu  Sat Apr 13 19:24:47 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Sun, 14 Apr 2019 00:24:47 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>
Message-ID: <DM6PR02MB524247B8D2C8F0FB71596E98D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>

I tried doing -ksp_view_mat binary, but I don't see the binaryoutput file being produced in my output or source file directories. Is it located somewhere else?

Best regards,
Yuyun

-----Original Message-----
From: Smith, Barry F. <bsmith at mcs.anl.gov> 
Sent: Thursday, April 11, 2019 10:21 PM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH


  Ok, still a little odd. PCSetOperators() which is called internally by KSPSetOperators() checks if the matrix has changed size and generates an error. Similar if you set a different matrix from before it resets the computation of the preconditioner. So, in theory, your situation should never occur.

   Barry


> On Apr 12, 2019, at 12:01 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I think this problem arose because I did not reset the ksp for solving 
> a different problem! It's not giving me an error anymore now that I 
> added the reset, so it's all good :)
> 
> Thanks,
> Yuyun
> 
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 9:21:11 PM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
> MPICH
>  
> 
>    Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
> 
>    Barry
> 
> 
> 
> > On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> > 
> > Thanks Barry for the detailed answers!
> > 
> > Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
> > 
> > Thanks again for the timely help!
> > Yuyun
> > 
> > Get Outlook for iOS
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Thursday, April 11, 2019 6:44:54 PM
> > To: Yuyun Yang
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Question about KSP, and makefile linking 
> > MPICH
> >  
> > 
> > 
> > > On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > Hello team,
> > >  
> > > I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
> > 
> >    Do you mean call KSPSetOperators() with one matrix and then later 
> > call it with a different matrix? This is ok if the two matrices are 
> > the same size and have the same parallel layout. But if the matrices 
> > are different size, have different parallel layout then you need to 
> > destroy the KSP and create a new one or call KSPReset() in between 
> > for example
> > 
> >   KSPSetFromOptions(ksp);
> >   KSPSetOperators(ksp,A,A);
> >   KSPSolve(ksp,b,x); 
> >   KSPReset(ksp);
> >   KSPSetOperators(ksp,B,B);
> >   KSPSolve(ksp,newb,newx);
> > 
> > >  
> > > Also, I know I?ve asked this before about linking to MPICH when I 
> > > call mpirun, instead of using my computer?s default MPI, but I 
> > > want to check again. The same problem was solved on my cluster by 
> > > using a different CLINKER (called mpiicc) in the Makefile and a 
> > > different intel compiler, which will link my compiled code with 
> > > MPICH. Is there a similar thing I can do on my own computer, 
> > > instead of having to use a very long path to locate the MPICH I 
> > > configured with PETSc, and then calling the executable? (I tried 
> > > making CLINKER = mpiicc on my own computer but that didn?t work.)
> > 
> >     Are you asking how you can avoid something like
> > 
> >       /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
> > 
> >    You can add /home/me/petsc/arch-myarch/bin to the beginning of 
> > your PATH, for example with bash put the following into your 
> > ~/.bashrc file
> > 
> >       export PATH=/home/me/petsc/arch-myarch/bin:$PATH
> >       mpiexec -n 2 ./mycode
> > 
> > >  
> > > The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
> > >  
> > > ==830== Invalid read of size 8
> > > ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
> > > ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
> > > ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
> > > ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
> > > ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
> > > ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > >  
> > > ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 
> > > alloc'd
> > >  
> > > ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > > ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
> > > ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
> > > ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
> > > ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
> > > ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
> > > ==830==    by 0x62BA574: PCSetUp (precon.c:932)
> > > ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
> > > ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
> > > ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
> > > ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
> > > ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
> > 
> >    This is curious. The line in the MUMPS code where valgrind 
> > detects a problem is
> > 
> >             K = 1_8
> >             THEMIN = ZERO
> >             DO
> >                IF(THEMIN .NE. ZERO) EXIT
> >                THEMIN = abs(id%A(K))                               <<<<<<< this line
> >                K = K+1_8
> > 
> >    So it has a problem accessing id%A(1)  the very first entry in 
> > numerical values of the sparse matrix. Meanwhile it states
> > > 0 bytes after a block of size 7,872 alloc'd 
> > > MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where 
> > > PETSc allocates
> > the values passed to MUMPS. So it almost as if MUMPS never allocated 
> > any space for id%A(), I can't imagine why that would ever happen 
> > (the problem size is super small so its not like it might have run 
> > out of memory)
> > 
> >     What happens if you allow the valgrind to continue? Do you get more valgrind errors?
> > 
> >     What happens if run without valgrind? Does it crash at this 
> > point in the code? At some later point? Does it run to completion 
> > and seem to produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look reasonable.
> > 
> >    Barry
> > 
> > 
> > 
> > 
> > >  
> > > Thank you!
> > > Yuyun


From bsmith at mcs.anl.gov  Sat Apr 13 21:07:45 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 14 Apr 2019 02:07:45 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB524247B8D2C8F0FB71596E98D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>
	<DM6PR02MB524247B8D2C8F0FB71596E98D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <6FACFDE5-8B5B-49A8-A0C5-335FF048030F@mcs.anl.gov>


  It will be in the directory where the program is run. 

  Perhaps you are not calling KSPSetFromOptions()? This is where it is checked.

   Barry


> On Apr 13, 2019, at 7:24 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I tried doing -ksp_view_mat binary, but I don't see the binaryoutput file being produced in my output or source file directories. Is it located somewhere else?
> 
> Best regards,
> Yuyun
> 
> -----Original Message-----
> From: Smith, Barry F. <bsmith at mcs.anl.gov> 
> Sent: Thursday, April 11, 2019 10:21 PM
> To: Yuyun Yang <yyang85 at stanford.edu>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
> 
> 
>  Ok, still a little odd. PCSetOperators() which is called internally by KSPSetOperators() checks if the matrix has changed size and generates an error. Similar if you set a different matrix from before it resets the computation of the preconditioner. So, in theory, your situation should never occur.
> 
>   Barry
> 
> 
>> On Apr 12, 2019, at 12:01 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>> 
>> I think this problem arose because I did not reset the ksp for solving 
>> a different problem! It's not giving me an error anymore now that I 
>> added the reset, so it's all good :)
>> 
>> Thanks,
>> Yuyun
>> 
>> Get Outlook for iOS
>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>> Sent: Thursday, April 11, 2019 9:21:11 PM
>> To: Yuyun Yang
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>> MPICH
>> 
>> 
>>   Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
>> 
>>   Barry
>> 
>> 
>> 
>>> On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>>> 
>>> Thanks Barry for the detailed answers!
>>> 
>>> Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
>>> 
>>> Thanks again for the timely help!
>>> Yuyun
>>> 
>>> Get Outlook for iOS
>>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>>> Sent: Thursday, April 11, 2019 6:44:54 PM
>>> To: Yuyun Yang
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>>> MPICH
>>> 
>>> 
>>> 
>>>> On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> Hello team,
>>>> 
>>>> I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
>>> 
>>>   Do you mean call KSPSetOperators() with one matrix and then later 
>>> call it with a different matrix? This is ok if the two matrices are 
>>> the same size and have the same parallel layout. But if the matrices 
>>> are different size, have different parallel layout then you need to 
>>> destroy the KSP and create a new one or call KSPReset() in between 
>>> for example
>>> 
>>>  KSPSetFromOptions(ksp);
>>>  KSPSetOperators(ksp,A,A);
>>>  KSPSolve(ksp,b,x); 
>>>  KSPReset(ksp);
>>>  KSPSetOperators(ksp,B,B);
>>>  KSPSolve(ksp,newb,newx);
>>> 
>>>> 
>>>> Also, I know I?ve asked this before about linking to MPICH when I 
>>>> call mpirun, instead of using my computer?s default MPI, but I 
>>>> want to check again. The same problem was solved on my cluster by 
>>>> using a different CLINKER (called mpiicc) in the Makefile and a 
>>>> different intel compiler, which will link my compiled code with 
>>>> MPICH. Is there a similar thing I can do on my own computer, 
>>>> instead of having to use a very long path to locate the MPICH I 
>>>> configured with PETSc, and then calling the executable? (I tried 
>>>> making CLINKER = mpiicc on my own computer but that didn?t work.)
>>> 
>>>    Are you asking how you can avoid something like
>>> 
>>>      /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
>>> 
>>>   You can add /home/me/petsc/arch-myarch/bin to the beginning of 
>>> your PATH, for example with bash put the following into your 
>>> ~/.bashrc file
>>> 
>>>      export PATH=/home/me/petsc/arch-myarch/bin:$PATH
>>>      mpiexec -n 2 ./mycode
>>> 
>>>> 
>>>> The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
>>>> 
>>>> ==830== Invalid read of size 8
>>>> ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
>>>> ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
>>>> ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
>>>> ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
>>>> ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
>>>> ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>> 
>>>> ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 
>>>> alloc'd
>>>> 
>>>> ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
>>>> ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
>>>> ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>> ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
>>>> ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
>>> 
>>>   This is curious. The line in the MUMPS code where valgrind 
>>> detects a problem is
>>> 
>>>            K = 1_8
>>>            THEMIN = ZERO
>>>            DO
>>>               IF(THEMIN .NE. ZERO) EXIT
>>>               THEMIN = abs(id%A(K))                               <<<<<<< this line
>>>               K = K+1_8
>>> 
>>>   So it has a problem accessing id%A(1)  the very first entry in 
>>> numerical values of the sparse matrix. Meanwhile it states
>>>> 0 bytes after a block of size 7,872 alloc'd 
>>>> MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where 
>>>> PETSc allocates
>>> the values passed to MUMPS. So it almost as if MUMPS never allocated 
>>> any space for id%A(), I can't imagine why that would ever happen 
>>> (the problem size is super small so its not like it might have run 
>>> out of memory)
>>> 
>>>    What happens if you allow the valgrind to continue? Do you get more valgrind errors?
>>> 
>>>    What happens if run without valgrind? Does it crash at this 
>>> point in the code? At some later point? Does it run to completion 
>>> and seem to produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look reasonable.
>>> 
>>>   Barry
>>> 
>>> 
>>> 
>>> 
>>>> 
>>>> Thank you!
>>>> Yuyun
> 


From yyang85 at stanford.edu  Sun Apr 14 00:00:49 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Sun, 14 Apr 2019 05:00:49 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <6FACFDE5-8B5B-49A8-A0C5-335FF048030F@mcs.anl.gov>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>
	<DM6PR02MB524247B8D2C8F0FB71596E98D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<6FACFDE5-8B5B-49A8-A0C5-335FF048030F@mcs.anl.gov>
Message-ID: <DM6PR02MB52420E044D9751F1E3DE7064D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>

I tried using another linear solver (via command line options) and the binaryoutput file was produced, but nothing was produced for MumpsCholesky. Does that mean it didn't even do a single linear solve?

Best regards,
Yuyun

-----Original Message-----
From: Smith, Barry F. <bsmith at mcs.anl.gov> 
Sent: Saturday, April 13, 2019 7:08 PM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH


  It will be in the directory where the program is run. 

  Perhaps you are not calling KSPSetFromOptions()? This is where it is checked.

   Barry


> On Apr 13, 2019, at 7:24 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I tried doing -ksp_view_mat binary, but I don't see the binaryoutput file being produced in my output or source file directories. Is it located somewhere else?
> 
> Best regards,
> Yuyun
> 
> -----Original Message-----
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Thursday, April 11, 2019 10:21 PM
> To: Yuyun Yang <yyang85 at stanford.edu>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
> MPICH
> 
> 
>  Ok, still a little odd. PCSetOperators() which is called internally by KSPSetOperators() checks if the matrix has changed size and generates an error. Similar if you set a different matrix from before it resets the computation of the preconditioner. So, in theory, your situation should never occur.
> 
>   Barry
> 
> 
>> On Apr 12, 2019, at 12:01 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>> 
>> I think this problem arose because I did not reset the ksp for 
>> solving a different problem! It's not giving me an error anymore now 
>> that I added the reset, so it's all good :)
>> 
>> Thanks,
>> Yuyun
>> 
>> Get Outlook for iOS
>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>> Sent: Thursday, April 11, 2019 9:21:11 PM
>> To: Yuyun Yang
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>> MPICH
>> 
>> 
>>   Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
>> 
>>   Barry
>> 
>> 
>> 
>>> On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>>> 
>>> Thanks Barry for the detailed answers!
>>> 
>>> Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
>>> 
>>> Thanks again for the timely help!
>>> Yuyun
>>> 
>>> Get Outlook for iOS
>>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>>> Sent: Thursday, April 11, 2019 6:44:54 PM
>>> To: Yuyun Yang
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>>> MPICH
>>> 
>>> 
>>> 
>>>> On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> Hello team,
>>>> 
>>>> I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
>>> 
>>>   Do you mean call KSPSetOperators() with one matrix and then later 
>>> call it with a different matrix? This is ok if the two matrices are 
>>> the same size and have the same parallel layout. But if the matrices 
>>> are different size, have different parallel layout then you need to 
>>> destroy the KSP and create a new one or call KSPReset() in between 
>>> for example
>>> 
>>>  KSPSetFromOptions(ksp);
>>>  KSPSetOperators(ksp,A,A);
>>>  KSPSolve(ksp,b,x);
>>>  KSPReset(ksp);
>>>  KSPSetOperators(ksp,B,B);
>>>  KSPSolve(ksp,newb,newx);
>>> 
>>>> 
>>>> Also, I know I?ve asked this before about linking to MPICH when I 
>>>> call mpirun, instead of using my computer?s default MPI, but I want 
>>>> to check again. The same problem was solved on my cluster by using 
>>>> a different CLINKER (called mpiicc) in the Makefile and a different 
>>>> intel compiler, which will link my compiled code with MPICH. Is 
>>>> there a similar thing I can do on my own computer, instead of 
>>>> having to use a very long path to locate the MPICH I configured 
>>>> with PETSc, and then calling the executable? (I tried making 
>>>> CLINKER = mpiicc on my own computer but that didn?t work.)
>>> 
>>>    Are you asking how you can avoid something like
>>> 
>>>      /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
>>> 
>>>   You can add /home/me/petsc/arch-myarch/bin to the beginning of 
>>> your PATH, for example with bash put the following into your 
>>> ~/.bashrc file
>>> 
>>>      export PATH=/home/me/petsc/arch-myarch/bin:$PATH
>>>      mpiexec -n 2 ./mycode
>>> 
>>>> 
>>>> The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
>>>> 
>>>> ==830== Invalid read of size 8
>>>> ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
>>>> ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
>>>> ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
>>>> ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
>>>> ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
>>>> ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>> 
>>>> ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 
>>>> alloc'd
>>>> 
>>>> ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
>>>> ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
>>>> ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>> ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
>>>> ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
>>> 
>>>   This is curious. The line in the MUMPS code where valgrind detects 
>>> a problem is
>>> 
>>>            K = 1_8
>>>            THEMIN = ZERO
>>>            DO
>>>               IF(THEMIN .NE. ZERO) EXIT
>>>               THEMIN = abs(id%A(K))                               <<<<<<< this line
>>>               K = K+1_8
>>> 
>>>   So it has a problem accessing id%A(1)  the very first entry in 
>>> numerical values of the sparse matrix. Meanwhile it states
>>>> 0 bytes after a block of size 7,872 alloc'd 
>>>> MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where 
>>>> PETSc allocates
>>> the values passed to MUMPS. So it almost as if MUMPS never allocated 
>>> any space for id%A(), I can't imagine why that would ever happen 
>>> (the problem size is super small so its not like it might have run 
>>> out of memory)
>>> 
>>>    What happens if you allow the valgrind to continue? Do you get more valgrind errors?
>>> 
>>>    What happens if run without valgrind? Does it crash at this point 
>>> in the code? At some later point? Does it run to completion and seem 
>>> to produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look reasonable.
>>> 
>>>   Barry
>>> 
>>> 
>>> 
>>> 
>>>> 
>>>> Thank you!
>>>> Yuyun
> 


From bsmith at mcs.anl.gov  Sun Apr 14 00:34:48 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 14 Apr 2019 05:34:48 +0000
Subject: [petsc-users] Question about KSP, and makefile linking MPICH
In-Reply-To: <DM6PR02MB52420E044D9751F1E3DE7064D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242154E1D70EFACA7AED296D22F0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<523BD840-AEA2-476E-AACF-D405D5C82760@anl.gov>
	<DM6PR02MB5242204EBF9FE9FC2E61E08DD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<645EC46C-EF5C-424B-9DE5-04463AD638F3@mcs.anl.gov>
	<DM6PR02MB52426E2DA69DEE75F4FEF1CBD2280@DM6PR02MB5242.namprd02.prod.outlook.com>
	<51346B5C-F1A3-4928-8DBB-F1EED95F164F@mcs.anl.gov>
	<DM6PR02MB524247B8D2C8F0FB71596E98D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<6FACFDE5-8B5B-49A8-A0C5-335FF048030F@mcs.anl.gov>
	<DM6PR02MB52420E044D9751F1E3DE7064D22A0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <0431FD39-24E9-4D43-96C7-A2ECB82A1451@mcs.anl.gov>



> On Apr 14, 2019, at 12:00 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I tried using another linear solver (via command line options) and the binaryoutput file was produced, but nothing was produced for MumpsCholesky. Does that mean it didn't even do a single linear solve?

   This is only a guess but since the program crashes in the MUMPS Cholesky case perhaps the program crashed before the file was closed so the file output was lost. 

   Anyways send the file output that was generated with the different solver (since presumably it is the same matrix as the MUMPS case) to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to crash on it.

  Barry


> 
> Best regards,
> Yuyun
> 
> -----Original Message-----
> From: Smith, Barry F. <bsmith at mcs.anl.gov> 
> Sent: Saturday, April 13, 2019 7:08 PM
> To: Yuyun Yang <yyang85 at stanford.edu>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Question about KSP, and makefile linking MPICH
> 
> 
>  It will be in the directory where the program is run. 
> 
>  Perhaps you are not calling KSPSetFromOptions()? This is where it is checked.
> 
>   Barry
> 
> 
>> On Apr 13, 2019, at 7:24 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>> 
>> I tried doing -ksp_view_mat binary, but I don't see the binaryoutput file being produced in my output or source file directories. Is it located somewhere else?
>> 
>> Best regards,
>> Yuyun
>> 
>> -----Original Message-----
>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>> Sent: Thursday, April 11, 2019 10:21 PM
>> To: Yuyun Yang <yyang85 at stanford.edu>
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>> MPICH
>> 
>> 
>> Ok, still a little odd. PCSetOperators() which is called internally by KSPSetOperators() checks if the matrix has changed size and generates an error. Similar if you set a different matrix from before it resets the computation of the preconditioner. So, in theory, your situation should never occur.
>> 
>>  Barry
>> 
>> 
>>> On Apr 12, 2019, at 12:01 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>>> 
>>> I think this problem arose because I did not reset the ksp for 
>>> solving a different problem! It's not giving me an error anymore now 
>>> that I added the reset, so it's all good :)
>>> 
>>> Thanks,
>>> Yuyun
>>> 
>>> Get Outlook for iOS
>>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>>> Sent: Thursday, April 11, 2019 9:21:11 PM
>>> To: Yuyun Yang
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>>> MPICH
>>> 
>>> 
>>>  Ahh, I just realized one other thing we can try. Run the program that crashes with -ksp_mat_view binary  this will produce a file called binaryoutput, send that file to petsc-maint at mcs.anl.gov and we'll see if we can get MUMPS to mis-behave with it also.
>>> 
>>>  Barry
>>> 
>>> 
>>> 
>>>> On Apr 11, 2019, at 11:17 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>>>> 
>>>> Thanks Barry for the detailed answers!
>>>> 
>>>> Regarding the problem with valgrind, this is the only error produced, and if I allow it to run further, the program would break (at a later function I get NaN for some of the values being calculated, and I've put an assert to prevent NaN results). I will take a look at it in the debugger. This is for testing, but for bigger problems I won't end up using Cholesky, so it's not really a big issue.
>>>> 
>>>> Thanks again for the timely help!
>>>> Yuyun
>>>> 
>>>> Get Outlook for iOS
>>>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>>>> Sent: Thursday, April 11, 2019 6:44:54 PM
>>>> To: Yuyun Yang
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] Question about KSP, and makefile linking 
>>>> MPICH
>>>> 
>>>> 
>>>> 
>>>>> On Apr 11, 2019, at 5:44 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> Hello team,
>>>>> 
>>>>> I?d like to check if it?s ok to use the same ksp object and change its operator (the matrix A) later on in the code to solve a different problem?
>>>> 
>>>>  Do you mean call KSPSetOperators() with one matrix and then later 
>>>> call it with a different matrix? This is ok if the two matrices are 
>>>> the same size and have the same parallel layout. But if the matrices 
>>>> are different size, have different parallel layout then you need to 
>>>> destroy the KSP and create a new one or call KSPReset() in between 
>>>> for example
>>>> 
>>>> KSPSetFromOptions(ksp);
>>>> KSPSetOperators(ksp,A,A);
>>>> KSPSolve(ksp,b,x);
>>>> KSPReset(ksp);
>>>> KSPSetOperators(ksp,B,B);
>>>> KSPSolve(ksp,newb,newx);
>>>> 
>>>>> 
>>>>> Also, I know I?ve asked this before about linking to MPICH when I 
>>>>> call mpirun, instead of using my computer?s default MPI, but I want 
>>>>> to check again. The same problem was solved on my cluster by using 
>>>>> a different CLINKER (called mpiicc) in the Makefile and a different 
>>>>> intel compiler, which will link my compiled code with MPICH. Is 
>>>>> there a similar thing I can do on my own computer, instead of 
>>>>> having to use a very long path to locate the MPICH I configured 
>>>>> with PETSc, and then calling the executable? (I tried making 
>>>>> CLINKER = mpiicc on my own computer but that didn?t work.)
>>>> 
>>>>   Are you asking how you can avoid something like
>>>> 
>>>>     /home/me/petsc/arch-myarch/bin/mpiexec -n 2 ./mycode ?
>>>> 
>>>>  You can add /home/me/petsc/arch-myarch/bin to the beginning of 
>>>> your PATH, for example with bash put the following into your 
>>>> ~/.bashrc file
>>>> 
>>>>     export PATH=/home/me/petsc/arch-myarch/bin:$PATH
>>>>     mpiexec -n 2 ./mycode
>>>> 
>>>>> 
>>>>> The final question is related to valgrind. I have defined a setupKSP function to do all the solver/pc setup. It seems like there is a problem with memory allocation but I don?t really understand why. This only happens for MUMPSCHOLESKY though (running CG, AMG etc. was fine):
>>>>> 
>>>>> ==830== Invalid read of size 8
>>>>> ==830==    at 0x6977C95: dmumps_ana_o_ (dana_aux.F:2054)
>>>>> ==830==    by 0x6913B5A: dmumps_ana_driver_ (dana_driver.F:390)
>>>>> ==830==    by 0x68C152C: dmumps_ (dmumps_driver.F:1213)
>>>>> ==830==    by 0x68BBE1C: dmumps_f77_ (dmumps_f77.F:267)
>>>>> ==830==    by 0x68BA4EB: dmumps_c (mumps_c.c:417)
>>>>> ==830==    by 0x5A070D6: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1654)
>>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>>> 
>>>>> ==830==  Address 0xb8149c0 is 0 bytes after a block of size 7,872 
>>>>> alloc'd
>>>>> 
>>>>> ==830==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==830==    by 0x500E7E0: PetscMallocAlign (mal.c:41)
>>>>> ==830==    by 0x59F8A16: MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)
>>>>> ==830==    by 0x5A06B53: MatCholeskyFactorSymbolic_MUMPS (mumps.c:1618)
>>>>> ==830==    by 0x54071F2: MatCholeskyFactorSymbolic (matrix.c:3179)
>>>>> ==830==    by 0x614AFE9: PCSetUp_Cholesky (cholesky.c:88)
>>>>> ==830==    by 0x62BA574: PCSetUp (precon.c:932)
>>>>> ==830==    by 0x640BB29: KSPSetUp (itfunc.c:391)
>>>>> ==830==    by 0x4A1192: PressureEq::setupKSP(_p_KSP*&, _p_PC*&, _p_Mat*&) (pressureEq.cpp:834)
>>>>> ==830==    by 0x4A1258: PressureEq::computeInitialSteadyStatePressure(Domain&) (pressureEq.cpp:862)
>>>>> ==830==    by 0x49B809: PressureEq::PressureEq(Domain&) (pressureEq.cpp:62)
>>>>> ==830==    by 0x4A88E9: StrikeSlip_LinearElastic_qd::StrikeSlip_LinearElastic_qd(Domain&) (strikeSlip_linearElastic_qd.cpp:57)
>>>> 
>>>>  This is curious. The line in the MUMPS code where valgrind detects 
>>>> a problem is
>>>> 
>>>>           K = 1_8
>>>>           THEMIN = ZERO
>>>>           DO
>>>>              IF(THEMIN .NE. ZERO) EXIT
>>>>              THEMIN = abs(id%A(K))                               <<<<<<< this line
>>>>              K = K+1_8
>>>> 
>>>>  So it has a problem accessing id%A(1)  the very first entry in 
>>>> numerical values of the sparse matrix. Meanwhile it states
>>>>> 0 bytes after a block of size 7,872 alloc'd 
>>>>> MatConvertToTriples_seqaij_seqsbaij (mumps.c:402)  which is where 
>>>>> PETSc allocates
>>>> the values passed to MUMPS. So it almost as if MUMPS never allocated 
>>>> any space for id%A(), I can't imagine why that would ever happen 
>>>> (the problem size is super small so its not like it might have run 
>>>> out of memory)
>>>> 
>>>>   What happens if you allow the valgrind to continue? Do you get more valgrind errors?
>>>> 
>>>>   What happens if run without valgrind? Does it crash at this point 
>>>> in the code? At some later point? Does it run to completion and seem 
>>>> to produce the correct answer? If it crashes, you could run it in the debugger and when it crashes print the value of id, id%A etc and see if they look reasonable.
>>>> 
>>>>  Barry
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> Thank you!
>>>>> Yuyun
>> 
> 


From sajidsyed2021 at u.northwestern.edu  Sun Apr 14 19:28:06 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Sun, 14 Apr 2019 19:28:06 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
Message-ID: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>

Hi PETSc Developers,

I happen to be a user who needs 4 vectors whose layout is aligned with
FFTW. The usual MatCreateVecsFFTW allows one to make 3 such vectors. To get
around this I call the function twice, once with three vectors, once with
one vector (and 2x NULL). This causes a strange segfault when freeing the
memory with VecDestroy (for the lone vector I'm guessing).

My program runs with no issues if I create the fourth vector with
VecCreateMPI (I need the 4th vector for point-wise multiply so this would
be ineffiient).

To get around this problem, is there a way to ask for 4 vectors aligned to
the FFTW matrix? If not is there a way to get the intended behavior from
VecCreateMPI, (perhaps by using a helper function to determine the data
alignment and pass to it instead of using PETSC_DECIDE)?

I'm attaching my code just in case what I'm thinking is wrong and anyone
would be kind enough to point it out to me. The issue is at line 87/88.
With 87, the program crashes, with 88 it works fine.

Thanks in advance for the help!

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/ba5f18f9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_ms.c
Type: application/octet-stream
Size: 9149 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/ba5f18f9/attachment.obj>

From knepley at gmail.com  Sun Apr 14 19:36:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 14 Apr 2019 20:36:16 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
Message-ID: <CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>

On Sun, Apr 14, 2019 at 8:29 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi PETSc Developers,
>
> I happen to be a user who needs 4 vectors whose layout is aligned with
> FFTW. The usual MatCreateVecsFFTW allows one to make 3 such vectors. To get
> around this I call the function twice, once with three vectors, once with
> one vector (and 2x NULL). This causes a strange segfault when freeing the
> memory with VecDestroy (for the lone vector I'm guessing).
>

1) These vectors have potentially different layout. That is why there are 3
arguments.

2) If you want a copy of one of these vectors, use VecDuplicate().

   Thanks,

     Matt


> My program runs with no issues if I create the fourth vector with
> VecCreateMPI (I need the 4th vector for point-wise multiply so this would
> be ineffiient).
>
> To get around this problem, is there a way to ask for 4 vectors aligned to
> the FFTW matrix? If not is there a way to get the intended behavior from
> VecCreateMPI, (perhaps by using a helper function to determine the data
> alignment and pass to it instead of using PETSC_DECIDE)?
>
> I'm attaching my code just in case what I'm thinking is wrong and anyone
> would be kind enough to point it out to me. The issue is at line 87/88.
> With 87, the program crashes, with 88 it works fine.
>
> Thanks in advance for the help!
>
> --
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/6050e7d9/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Sun Apr 14 19:59:39 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Sun, 14 Apr 2019 19:59:39 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
Message-ID: <CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>

 Hi Matt,

While in theory, that sounds perfect I still get the same error. I'm
attaching a minimal test program which creates 3 vectors x,y,z via the
petsc-fftw interface and a test vector via VecDuplicate and then destroy
all the vectors. Without the test vector everything works fine.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/9d192b23/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_modify.c
Type: application/octet-stream
Size: 1344 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/9d192b23/attachment.obj>

From knepley at gmail.com  Sun Apr 14 20:07:28 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 14 Apr 2019 21:07:28 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
Message-ID: <CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>

On Sun, Apr 14, 2019 at 9:00 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Hi Matt,
>
> While in theory, that sounds perfect I still get the same error. I'm
> attaching a minimal test program which creates 3 vectors x,y,z via the
> petsc-fftw interface and a test vector via VecDuplicate and then destroy
> all the vectors. Without the test vector everything works fine.
>

Runs fine for me:

master *:~/Downloads/tmp$ ./testfft

Use PETSc-FFTW interface...2-DIM: 256

Send the error.

    Matt

Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/8953874e/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Sun Apr 14 20:11:55 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Sun, 14 Apr 2019 20:11:55 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
Message-ID: <CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>

Just to confirm, there's no error when running with one rank. The error
occurs only with mpirun -np x (x>1).

Attaching the error log.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/99b853dd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out
Type: application/octet-stream
Size: 185444 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/99b853dd/attachment-0001.obj>

From knepley at gmail.com  Sun Apr 14 20:28:45 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 14 Apr 2019 21:28:45 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
Message-ID: <CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>

On Sun, Apr 14, 2019 at 9:12 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Just to confirm, there's no error when running with one rank. The error
> occurs only with mpirun -np x (x>1).
>

This is completely broken. I attached a version that will work in parallel,
but its ugly.

PETSc People:
The MatCreateVecsFFT()  calls fftw_malloc()!!! in parallel. What possible
motivation could there be?
This causes a failure because the custom destroy calls fftw_free().
VecDuplicate calls PetscMalloc(),
but then the custom destroy calls fftw_free() on that thing and chokes on
the header we put on all
allocated blocks. Its not easy to see who wrote the fftw_malloc() lines,
but it seems to be at least 8 years
ago. I can convert them to PetscMalloc(), but do we have any tests that
would make us confident that
this is not wrecking something? Is anyone familiar with this part of the
code?

  Matt


> Attaching the error log.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/d543ab49/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_modify.c
Type: application/octet-stream
Size: 1407 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/d543ab49/attachment.obj>

From sajidsyed2021 at u.northwestern.edu  Sun Apr 14 20:43:57 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Sun, 14 Apr 2019 20:43:57 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
Message-ID: <CAOGsD9h_D485yLgR=LOuyJ69Qa2znki_i0VNjdfM10SdSua4SA@mail.gmail.com>

Thanks for the temporary fix.

(PS: I was wondering if it would be trivial to just extend the code to have
four mallocs and create a new function but it looks like the logic is much
more complicated.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/da6baacc/attachment.html>

From knepley at gmail.com  Sun Apr 14 21:10:49 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 14 Apr 2019 22:10:49 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9h_D485yLgR=LOuyJ69Qa2znki_i0VNjdfM10SdSua4SA@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<CAOGsD9h_D485yLgR=LOuyJ69Qa2znki_i0VNjdfM10SdSua4SA@mail.gmail.com>
Message-ID: <CAMYG4G=qMwU_8cQ0D20ADYL3M2maNo+Q7Wffjzv3PBsmy4oZaA@mail.gmail.com>

On Sun, Apr 14, 2019 at 9:44 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Thanks for the temporary fix.
>
> (PS: I was wondering if it would be trivial to just extend the code to
> have four mallocs and create a new function but it looks like the logic is
> much more complicated.)
>

You would never ever ever extend this to have 4 vectors, because there
would be no meaning to the 4th vector. These
3 vectors have specific meanings, and can have different parallel layouts.

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/dcbc9e70/attachment.html>

From bsmith at mcs.anl.gov  Sun Apr 14 23:57:39 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 15 Apr 2019 04:57:39 +0000
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
Message-ID: <1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>


  http://www.fftw.org/doc/Memory-Allocation.html The issue isn't really the fftw_malloc() it is that the current code is broken if the user calls VecDuplicate() because the new vectors don't have the correct extra space needed by FFTW and could cause random crashes and incorrect answers. 

  The correct fix is to have a VecDuplicate_FFTW_fin() VecDuplicate_FFTW_fout() VecDuplicate_FFTW_bout(). Have MatCreateVecsFFTW_FFTW() 
attach the matrix to the new vector with PetscObjectCompose() then the duplicate routines would call PetscObjectQuery to get the matrix and then call 
MatCreateVecsFFTW_FFTW() for the duplicate.

   Pretty easy: Sajid do you want to give it a try?

But note: one has to be very careful what vectors they duplicate and that they use them in the right places; for example duplicating a fin vector but using it in an fout or bout location and there is no error checking for that in PETSc or FFT. To add error checking one could attach to the FFT vectors a marker indicating if it is fin, fout, bout and then calls to MatMult_FFTW() etc would check the markers on the vectors and error if they are incompatible.

    Barry

I find this FFTW model of requiring extra space in the arrays to be a horrifically fragile API. Perhaps I misunderstand it.



> On Apr 14, 2019, at 8:28 PM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Sun, Apr 14, 2019 at 9:12 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> Just to confirm, there's no error when running with one rank. The error occurs only with mpirun -np x (x>1). 
> 
> This is completely broken. I attached a version that will work in parallel, but its ugly.
> 
> PETSc People:
> The MatCreateVecsFFT()  calls fftw_malloc()!!! in parallel. What possible motivation could there be?
> This causes a failure because the custom destroy calls fftw_free(). VecDuplicate calls PetscMalloc(),
> but then the custom destroy calls fftw_free() on that thing and chokes on the header we put on all
> allocated blocks. Its not easy to see who wrote the fftw_malloc() lines, but it seems to be at least 8 years
> ago. I can convert them to PetscMalloc(), but do we have any tests that would make us confident that
> this is not wrecking something? Is anyone familiar with this part of the code?
> 
>   Matt
>  
> Attaching the error log.
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <ex_modify.c>


From hong at aspiritech.org  Sun Apr 14 23:05:20 2019
From: hong at aspiritech.org (hong at aspiritech.org)
Date: Sun, 14 Apr 2019 23:05:20 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4G=qMwU_8cQ0D20ADYL3M2maNo+Q7Wffjzv3PBsmy4oZaA@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<CAOGsD9h_D485yLgR=LOuyJ69Qa2znki_i0VNjdfM10SdSua4SA@mail.gmail.com>
	<CAMYG4G=qMwU_8cQ0D20ADYL3M2maNo+Q7Wffjzv3PBsmy4oZaA@mail.gmail.com>
Message-ID: <CAGCphBt3_F=O-pfGB8vemL+J_jsnFAfz8Sg0RDCxGBhp6MysUA@mail.gmail.com>

FFTW interface was written by a summer graduate student many years ago. It
was only used occasionally by users. Shall we continue maintain it? If so,
it needs to be updated. For users, using FFTW directly might be better
instead of using this interface.
Hong

On Sun, Apr 14, 2019 at 9:12 PM Matthew Knepley via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> On Sun, Apr 14, 2019 at 9:44 PM Sajid Ali <
> sajidsyed2021 at u.northwestern.edu> wrote:
>
>> Thanks for the temporary fix.
>>
>> (PS: I was wondering if it would be trivial to just extend the code to
>> have four mallocs and create a new function but it looks like the logic is
>> much more complicated.)
>>
>
> You would never ever ever extend this to have 4 vectors, because there
> would be no meaning to the 4th vector. These
> 3 vectors have specific meanings, and can have different parallel layouts.
>
>   Thanks,
>
>      Matt
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190414/7f35f761/attachment.html>

From knepley at gmail.com  Mon Apr 15 06:11:57 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Apr 2019 07:11:57 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
Message-ID: <CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>

On Mon, Apr 15, 2019 at 12:57 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   http://www.fftw.org/doc/Memory-Allocation.html The issue isn't really
> the fftw_malloc() it is that the current code is broken if the user calls
> VecDuplicate() because the new vectors don't have the correct extra space
> needed by FFTW and could cause random crashes and incorrect answers.
>
>   The correct fix is to have a VecDuplicate_FFTW_fin()
> VecDuplicate_FFTW_fout() VecDuplicate_FFTW_bout(). Have
> MatCreateVecsFFTW_FFTW()
> attach the matrix to the new vector with PetscObjectCompose() then the
> duplicate routines would call PetscObjectQuery to get the matrix and then
> call
> MatCreateVecsFFTW_FFTW() for the duplicate.
>
>    Pretty easy: Sajid do you want to give it a try?
>
> But note: one has to be very careful what vectors they duplicate and that
> they use them in the right places; for example duplicating a fin vector but
> using it in an fout or bout location and there is no error checking for
> that in PETSc or FFT. To add error checking one could attach to the FFT
> vectors a marker indicating if it is fin, fout, bout and then calls to
> MatMult_FFTW() etc would check the markers on the vectors and error if they
> are incompatible.
>
>     Barry
>
> I find this FFTW model of requiring extra space in the arrays to be a
> horrifically fragile API. Perhaps I misunderstand it.
>

I read their documentation differently, so maybe we should discuss this.
The only thing their malloc does is alignment. We do the
padding ourselves, so we could just switch all those statements to
PetscMalloc(). The only potential downside is them aligning
on a bigger offset, which I doubt because we do the same kind of
vectorization.

   Matt


>
>
> > On Apr 14, 2019, at 8:28 PM, Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > On Sun, Apr 14, 2019 at 9:12 PM Sajid Ali <
> sajidsyed2021 at u.northwestern.edu> wrote:
> > Just to confirm, there's no error when running with one rank. The error
> occurs only with mpirun -np x (x>1).
> >
> > This is completely broken. I attached a version that will work in
> parallel, but its ugly.
> >
> > PETSc People:
> > The MatCreateVecsFFT()  calls fftw_malloc()!!! in parallel. What
> possible motivation could there be?
> > This causes a failure because the custom destroy calls fftw_free().
> VecDuplicate calls PetscMalloc(),
> > but then the custom destroy calls fftw_free() on that thing and chokes
> on the header we put on all
> > allocated blocks. Its not easy to see who wrote the fftw_malloc() lines,
> but it seems to be at least 8 years
> > ago. I can convert them to PetscMalloc(), but do we have any tests that
> would make us confident that
> > this is not wrecking something? Is anyone familiar with this part of the
> code?
> >
> >   Matt
> >
> > Attaching the error log.
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> > <ex_modify.c>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/f3e07ffb/attachment-0001.html>

From myriam.peyrounette at idris.fr  Mon Apr 15 07:45:15 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Mon, 15 Apr 2019 14:45:15 +0200
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
	<CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
Message-ID: <26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>

Hi,

you'll find the new scaling attached (green line). I used the version
3.11 and the four scalability options :
-matptap_via scalable
-inner_diag_matmatmult_via scalable
-inner_offdiag_matmatmult_via scalable
-mat_freeintermediatedatastructures

The scaling is much better! The code even uses less memory for the
smallest cases. There is still an increase for the larger one.

With regard to the time scaling, I used KSPView and LogView on the two
previous scalings (blue and yellow lines) but not on the last one (green
line). So we can't really compare them, am I right? However, we can see
that the new time scaling looks quite good. It slightly increases from
~8s to ~27s.

Unfortunately, the computations are expensive so I would like to avoid
re-run them if possible. How relevant would be a proper time scaling for
you??

Myriam


Le 04/12/19 ? 18:18, Zhang, Hong a ?crit?:
> Myriam :
> Thanks for your effort. It will help us improve PETSc.
> Hong
>
>     Hi all,
>
>     I used the wrong script, that's why it diverged... Sorry about that.?
>     I tried again with the right script applied on a tiny problem (~200
>     elements). I can see a small difference in memory usage (gain ~ 1mB).
>     when adding the -mat_freeintermediatestructures option. I still
>     have to
>     execute larger cases to plot the scaling. The supercomputer I am
>     used to
>     run my jobs on is really busy at the moment so it takes a while. I
>     hope
>     I'll send you the results on Monday.
>
>     Thanks everyone,
>
>     Myriam
>
>
>     Le 04/11/19 ? 06:01, Jed Brown a ?crit?:
>     > "Zhang, Hong" <hzhang at mcs.anl.gov <mailto:hzhang at mcs.anl.gov>>
>     writes:
>     >
>     >> Jed:
>     >>>> Myriam,
>     >>>> Thanks for the plot. '-mat_freeintermediatedatastructures'
>     should not affect solution. It releases almost half of memory in
>     C=PtAP if C is not reused.
>     >>> And yet if turning it on causes divergence, that would imply a
>     bug.
>     >>> Hong, are you able to reproduce the experiment to see the memory
>     >>> scaling?
>     >> I like to test his code using an alcf machine, but my hands are
>     full now. I'll try it as soon as I find time, hopefully next week.
>     > I have now compiled and run her code locally.
>     >
>     > Myriam, thanks for your last mail adding configuration and
>     removing the
>     > MemManager.h dependency.? I ran with and without
>     > -mat_freeintermediatedatastructures and don't see a difference in
>     > convergence.? What commands did you run to observe that difference?
>
>     -- 
>     Myriam Peyrounette
>     CNRS/IDRIS - HLST
>     --
>
>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4540a0e6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem_scaling_big_cases_ex42_irene_SCALABLE.png
Type: image/png
Size: 35366 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4540a0e6/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: time_scaling_big_cases_ex42_irene_SCALABLE.png
Type: image/png
Size: 22181 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4540a0e6/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4540a0e6/attachment-0001.p7s>

From sajidsyed2021 at u.northwestern.edu  Mon Apr 15 11:47:24 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 15 Apr 2019 11:47:24 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
Message-ID: <CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>

Hi Barry & Matt,

I'd be happy to contribute a patch once I understand what's going on.

@Matt, Where is the padding occurring? In the VecCreateFFTW I see that each
process looks up the dimension of array it's supposed to hold and asks for
memory to hold that via fftw_malloc (which as you say is just a wrapper to
simd-aligned malloc). Is the crash occurring because the first vector was
created via fftw_malloc and duplicated via PETScMalloc and they happen to
have different alignment sizes (FFTW was compiled with simd=avx2 since I'm
using a Broadwell-Xeon and PETScMalloc aligns to PETSC_MEMALIGN ?)

PS: I've only ever used FFTW via the python interface (and automated the
build & but couldn't automate testing of pyfftw-mpi since cython coverage
reporting is confusing).

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4aa947ae/attachment.html>

From knepley at gmail.com  Mon Apr 15 11:53:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Apr 2019 12:53:04 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
Message-ID: <CAMYG4Gmdt9U4PkL7e9y-yWXuFNNrpAqit8DFqq4JEMuLjMrSdw@mail.gmail.com>

On Mon, Apr 15, 2019 at 12:48 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

>
> Hi Barry & Matt,
>
> I'd be happy to contribute a patch once I understand what's going on.
>
> @Matt, Where is the padding occurring? In the VecCreateFFTW I see that
> each process looks up the dimension of array it's supposed to hold and asks
> for memory to hold that via fftw_malloc (which as you say is just a wrapper
> to simd-aligned malloc). Is the crash occurring because the first vector
> was created via fftw_malloc and duplicated via PETScMalloc and they happen
> to have different alignment sizes (FFTW was compiled with simd=avx2 since
> I'm using a Broadwell-Xeon and PETScMalloc aligns to PETSC_MEMALIGN ?)
>

No. What is happening is that PetscMalloc() overallocates, writes a little
header on the the block of memory, and returns a pointer which is insider
the block after the header. However, this pointer was passed to fftw_free()
which just calls free() in it. free() says it never heard of this pointer
because its inside the block that was actually allocated (this is what
valgrind is telling us).

The change that is needed is just to replace all fftw_malloc() by
PetscMalloc1(). You have to be a little careful about
types to make sure we get the right sizes, but its not bad. I would do it
if this were not the end of the semester.

  Thanks,

    Matt


> PS: I've only ever used FFTW via the python interface (and automated the
> build & but couldn't automate testing of pyfftw-mpi since cython coverage
> reporting is confusing).
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/0da6ffb5/attachment.html>

From hzhang at mcs.anl.gov  Mon Apr 15 12:10:29 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Mon, 15 Apr 2019 17:10:29 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
	<CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
	<26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>
Message-ID: <CAGCphBvNdLMfZhrXDBtdgzaikujQu98WarT6Qr9LGF5i0AAqUA@mail.gmail.com>

Myriam:
Thank you very much for providing these results!
I have put effort to accelerate execution time and avoid using global sizes in PtAP, for which the algorithm of transpose of P_local and P_other likely doubles the memory usage. I'll try to investigate why it becomes unscalable.
Hong

Hi,

you'll find the new scaling attached (green line). I used the version 3.11 and the four scalability options :
-matptap_via scalable
-inner_diag_matmatmult_via scalable
-inner_offdiag_matmatmult_via scalable
-mat_freeintermediatedatastructures

The scaling is much better! The code even uses less memory for the smallest cases. There is still an increase for the larger one.

With regard to the time scaling, I used KSPView and LogView on the two previous scalings (blue and yellow lines) but not on the last one (green line). So we can't really compare them, am I right? However, we can see that the new time scaling looks quite good. It slightly increases from ~8s to ~27s.

Unfortunately, the computations are expensive so I would like to avoid re-run them if possible. How relevant would be a proper time scaling for you?

Myriam

Le 04/12/19 ? 18:18, Zhang, Hong a ?crit :
Myriam :
Thanks for your effort. It will help us improve PETSc.
Hong

Hi all,

I used the wrong script, that's why it diverged... Sorry about that.
I tried again with the right script applied on a tiny problem (~200
elements). I can see a small difference in memory usage (gain ~ 1mB).
when adding the -mat_freeintermediatestructures option. I still have to
execute larger cases to plot the scaling. The supercomputer I am used to
run my jobs on is really busy at the moment so it takes a while. I hope
I'll send you the results on Monday.

Thanks everyone,

Myriam


Le 04/11/19 ? 06:01, Jed Brown a ?crit :
> "Zhang, Hong" <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> writes:
>
>> Jed:
>>>> Myriam,
>>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
>>> And yet if turning it on causes divergence, that would imply a bug.
>>> Hong, are you able to reproduce the experiment to see the memory
>>> scaling?
>> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.
> I have now compiled and run her code locally.
>
> Myriam, thanks for your last mail adding configuration and removing the
> MemManager.h dependency.  I ran with and without
> -mat_freeintermediatedatastructures and don't see a difference in
> convergence.  What commands did you run to observe that difference?

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--




--
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/4c9d651d/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr 15 13:44:14 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 15 Apr 2019 18:44:14 +0000
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
Message-ID: <C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>


  There are two distinct issues here. 

1) the use of fftw_malloc(). This is a relatively minor issue. This is causing the crash in the code because VecDuplicate() uses PetscMalloc() to obtain the array but when the array is freed fftw_malloc() is called on it. 

2) the padding that FFTW needs in its vectors. Look at MatCreateVecsFFTW_FFTW() source code

      alloc_local = fftw_mpi_local_size_1d(dim[0],comm,FFTW_FORWARD,FFTW_ESTIMATE,&local_n0,&local_0_start,&local_n1,&local_1_start);
      if (fin) {
        data_fin  = (fftw_complex*)fftw_malloc(sizeof(fftw_complex)*alloc_local);
        ierr      = VecCreateMPIWithArray(comm,1,local_n0,N,(const PetscScalar*)data_fin,fin);CHKERRQ(ierr);

Note that fftw_mpi_local_size_1d returns a "size" alloc_local  that is then passed into the fftw_malloc() which is then passed into VecCreateMPIWithArray(); this size is LARGER than local_n0 the local size of the MPI vector.  FFTW routines assume that "extra space" is available at the end of the arrays and use that as "work" space. The different fin, fout, bout have sometimes different local sizes and different amount of padding (see the code for exact details). 

When you call VecDuplicate() on an fin vector it doesn't know about the padding so creates a new vector with an array with only the size of local_n0, that is without any ghost padding in the end. If you pass this new vector into a FFTW routine it will access out of bounds memory and thus potentially crash or incorrect results.  

Note that if one replaced all the fftw_malloc() with PetscMalloc() the error described above would still be there. VecDuplicate() vectors would not have the extra padding they need.

My proposed fixed outlined before resolves both problems at the same time.

   Barry

Final note just to make it absolutely clear, the use of fftw_malloc() has nothing to with the extra padding at the end of the vector arrays! The extra padding is managed by the call to alloc_local = fftw_mpi_local_size_1d() and the use of alloc_local to determine the size of the array to allocate. fftw_alloc() itself is not putting any extra padding at the end.






> On Apr 15, 2019, at 11:47 AM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> 
> Hi Barry & Matt, 
> 
> I'd be happy to contribute a patch once I understand what's going on. 
> 
> @Matt, Where is the padding occurring? In the VecCreateFFTW I see that each process looks up the dimension of array it's supposed to hold and asks for memory to hold that via fftw_malloc (which as you say is just a wrapper to simd-aligned malloc). Is the crash occurring because the first vector was created via fftw_malloc and duplicated via PETScMalloc and they happen to have different alignment sizes (FFTW was compiled with simd=avx2 since I'm using a Broadwell-Xeon and PETScMalloc aligns to PETSC_MEMALIGN ?)
> 
> PS: I've only ever used FFTW via the python interface (and automated the build & but couldn't automate testing of pyfftw-mpi since cython coverage reporting is confusing).  
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University


From xianghuang at gmail.com  Mon Apr 15 16:14:01 2019
From: xianghuang at gmail.com (Xiang Huang)
Date: Mon, 15 Apr 2019 16:14:01 -0500
Subject: [petsc-users] Link broken for: PETSc users manual - pdf (fully
 searchable with hyperlinks)
Message-ID: <6b52855b-4932-c523-f966-57be2d1cae16@gmail.com>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf

Error 404
The web server cannot find the file for which you are looking.

Best,
Xiang

From balay at mcs.anl.gov  Mon Apr 15 16:22:55 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 15 Apr 2019 21:22:55 +0000
Subject: [petsc-users] Link broken for: PETSc users manual - pdf (fully
 searchable with hyperlinks)
In-Reply-To: <6b52855b-4932-c523-f966-57be2d1cae16@gmail.com>
References: <6b52855b-4932-c523-f966-57be2d1cae16@gmail.com>
Message-ID: <alpine.LFD.2.21.1904151622110.2388@sb>

Looks like the manual didn't get generated correctly for petsc-3.11.1.

For now - I've restored the 3.11 manual - so the URL should work now.

Satish

On Mon, 15 Apr 2019, Xiang Huang via petsc-users wrote:

> http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
> 
> Error 404
> The web server cannot find the file for which you are looking.
> 
> Best,
> Xiang
> 


From knepley at gmail.com  Mon Apr 15 18:57:54 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Apr 2019 19:57:54 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
Message-ID: <CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>

On Mon, Apr 15, 2019 at 2:44 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   There are two distinct issues here.
>
> 1) the use of fftw_malloc(). This is a relatively minor issue. This is
> causing the crash in the code because VecDuplicate() uses PetscMalloc() to
> obtain the array but when the array is freed fftw_malloc() is called on it.
>
> 2) the padding that FFTW needs in its vectors. Look at
> MatCreateVecsFFTW_FFTW() source code
>
>       alloc_local =
> fftw_mpi_local_size_1d(dim[0],comm,FFTW_FORWARD,FFTW_ESTIMATE,&local_n0,&local_0_start,&local_n1,&local_1_start);
>       if (fin) {
>         data_fin  =
> (fftw_complex*)fftw_malloc(sizeof(fftw_complex)*alloc_local);
>         ierr      = VecCreateMPIWithArray(comm,1,local_n0,N,(const
> PetscScalar*)data_fin,fin);CHKERRQ(ierr);
>
> Note that fftw_mpi_local_size_1d returns a "size" alloc_local  that is
> then passed into the fftw_malloc() which is then passed into
> VecCreateMPIWithArray(); this size is LARGER than local_n0 the local size
> of the MPI vector.  FFTW routines assume that "extra space" is available at
> the end of the arrays and use that as "work" space. The different fin,
> fout, bout have sometimes different local sizes and different amount of
> padding (see the code for exact details).
>
> When you call VecDuplicate() on an fin vector it doesn't know about the
> padding so creates a new vector with an array with only the size of
> local_n0, that is without any ghost padding in the end. If you pass this
> new vector into a FFTW routine it will access out of bounds memory and thus
> potentially crash or incorrect results.
>
> Note that if one replaced all the fftw_malloc() with PetscMalloc() the
> error described above would still be there. VecDuplicate() vectors would
> not have the extra padding they need.
>
> My proposed fixed outlined before resolves both problems at the same time.
>

But its complicated. What about making VecGhost for this? VecDuplicate
respects the ghost region.

   Matt


>    Barry
>
> Final note just to make it absolutely clear, the use of fftw_malloc() has
> nothing to with the extra padding at the end of the vector arrays! The
> extra padding is managed by the call to alloc_local =
> fftw_mpi_local_size_1d() and the use of alloc_local to determine the size
> of the array to allocate. fftw_alloc() itself is not putting any extra
> padding at the end.
>
>
>
>
>
>
> > On Apr 15, 2019, at 11:47 AM, Sajid Ali <
> sajidsyed2021 at u.northwestern.edu> wrote:
> >
> >
> > Hi Barry & Matt,
> >
> > I'd be happy to contribute a patch once I understand what's going on.
> >
> > @Matt, Where is the padding occurring? In the VecCreateFFTW I see that
> each process looks up the dimension of array it's supposed to hold and asks
> for memory to hold that via fftw_malloc (which as you say is just a wrapper
> to simd-aligned malloc). Is the crash occurring because the first vector
> was created via fftw_malloc and duplicated via PETScMalloc and they happen
> to have different alignment sizes (FFTW was compiled with simd=avx2 since
> I'm using a Broadwell-Xeon and PETScMalloc aligns to PETSC_MEMALIGN ?)
> >
> > PS: I've only ever used FFTW via the python interface (and automated the
> build & but couldn't automate testing of pyfftw-mpi since cython coverage
> reporting is confusing).
> >
> > Thank You,
> > Sajid Ali
> > Applied Physics
> > Northwestern University
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190415/cc75c76d/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Apr 15 21:30:12 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 16 Apr 2019 02:30:12 +0000
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
Message-ID: <7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>



> On Apr 15, 2019, at 6:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Mon, Apr 15, 2019 at 2:44 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   There are two distinct issues here. 
> 
> 1) the use of fftw_malloc(). This is a relatively minor issue. This is causing the crash in the code because VecDuplicate() uses PetscMalloc() to obtain the array but when the array is freed fftw_malloc() is called on it. 
> 
> 2) the padding that FFTW needs in its vectors. Look at MatCreateVecsFFTW_FFTW() source code
> 
>       alloc_local = fftw_mpi_local_size_1d(dim[0],comm,FFTW_FORWARD,FFTW_ESTIMATE,&local_n0,&local_0_start,&local_n1,&local_1_start);
>       if (fin) {
>         data_fin  = (fftw_complex*)fftw_malloc(sizeof(fftw_complex)*alloc_local);
>         ierr      = VecCreateMPIWithArray(comm,1,local_n0,N,(const PetscScalar*)data_fin,fin);CHKERRQ(ierr);
> 
> Note that fftw_mpi_local_size_1d returns a "size" alloc_local  that is then passed into the fftw_malloc() which is then passed into VecCreateMPIWithArray(); this size is LARGER than local_n0 the local size of the MPI vector.  FFTW routines assume that "extra space" is available at the end of the arrays and use that as "work" space. The different fin, fout, bout have sometimes different local sizes and different amount of padding (see the code for exact details). 
> 
> When you call VecDuplicate() on an fin vector it doesn't know about the padding so creates a new vector with an array with only the size of local_n0, that is without any ghost padding in the end. If you pass this new vector into a FFTW routine it will access out of bounds memory and thus potentially crash or incorrect results.  
> 
> Note that if one replaced all the fftw_malloc() with PetscMalloc() the error described above would still be there. VecDuplicate() vectors would not have the extra padding they need.
> 
> My proposed fixed outlined before resolves both problems at the same time.
> 
> But its complicated.

   No it isn't. It's just proper subclassing of current MPI vector class. Nothing nonstandard at all.

> What about making VecGhost for this? VecDuplicate respects the ghost region.

   Possible but VecGhost has all the scatter information that doesn't exist for this vector subclass.

   Barry

> 
>    Matt
>  
>    Barry
> 
> Final note just to make it absolutely clear, the use of fftw_malloc() has nothing to with the extra padding at the end of the vector arrays! The extra padding is managed by the call to alloc_local = fftw_mpi_local_size_1d() and the use of alloc_local to determine the size of the array to allocate. fftw_alloc() itself is not putting any extra padding at the end.
> 
> 
> 
> 
> 
> 
> > On Apr 15, 2019, at 11:47 AM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> > 
> > 
> > Hi Barry & Matt, 
> > 
> > I'd be happy to contribute a patch once I understand what's going on. 
> > 
> > @Matt, Where is the padding occurring? In the VecCreateFFTW I see that each process looks up the dimension of array it's supposed to hold and asks for memory to hold that via fftw_malloc (which as you say is just a wrapper to simd-aligned malloc). Is the crash occurring because the first vector was created via fftw_malloc and duplicated via PETScMalloc and they happen to have different alignment sizes (FFTW was compiled with simd=avx2 since I'm using a Broadwell-Xeon and PETScMalloc aligns to PETSC_MEMALIGN ?)
> > 
> > PS: I've only ever used FFTW via the python interface (and automated the build & but couldn't automate testing of pyfftw-mpi since cython coverage reporting is confusing).  
> > 
> > Thank You,
> > Sajid Ali
> > Applied Physics
> > Northwestern University
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From yyang85 at stanford.edu  Tue Apr 16 01:14:07 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Tue, 16 Apr 2019 06:14:07 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
Message-ID: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I'm trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don't really tell me where the problems occur, for example:

[ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
      [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
      [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
      [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
      [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c

The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don't seem to be related to the above message (or maybe they are?).

How can I use this option to debug effectively?

Thanks a lot,
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/e44bbc6e/attachment.html>

From mfadams at lbl.gov  Tue Apr 16 03:04:02 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 16 Apr 2019 04:04:02 -0400
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CADOhEh5Rkq661TPnNwAwbmFDpN5ENH2U3vp8xiOcVdT3aUjBVw@mail.gmail.com>

Use valgrind with --leak-check=yes

This should give a stack trace at the end of the run.

On Tue, Apr 16, 2019 at 2:14 AM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello team,
>
>
>
> I?m trying to use the options -malloc_dump and -malloc_debug to examine
> memory leaks. The messages however, are quite generic, and don?t really
> tell me where the problems occur, for example:
>
>
>
> [ 0]1520 bytes VecCreate() line 35 in
> /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>
>       [0]  PetscMallocA() line 35 in
> /home/yyy910805/petsc/src/sys/memory/mal.c
>
>       [0]  VecCreate() line 30 in
> /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>
>       [0]  VecDuplicate_Seq() line 804 in
> /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
>
>       [0]  VecDuplicate() line 375 in
> /home/yyy910805/petsc/src/vec/vec/interface/vector.c
>
>
>
> The code is huge, so going through every single VecCreate/VecDuplicate and
> VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave
> me some uninitialized values errors that don?t seem to be related to the
> above message (or maybe they are?).
>
>
>
> How can I use this option to debug effectively?
>
>
>
> Thanks a lot,
>
> Yuyun
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/cc25c8ec/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr 16 07:54:15 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 16 Apr 2019 12:54:15 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>


  Please try the flag -options_dump this tries to give a much more concise view of what objects have not been freed. For example I commented
out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:

./ex19 -objects_dump
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
The following objects were never freed
-----------------------------------------
[0] DM da DM_0x84000000_0
      [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
[0] Vec seq Vec_0x84000000_6
      [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c

Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free. 

Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
For example because the unfreed vector holds a reference to the DM the DM is listed as not freed. Once you properly destroy the vector you'll 
not that the DM is no longer listed as non freed.

It can be a little overwhelming at first to figure out what objects have not been freed. We recommending setting the environmental variable
export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from
the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.

   Good luck



   Barry


> On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello team,
>  
> I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
>  
> [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
>       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
>       [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c
>  
> The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
>  
> How can I use this option to debug effectively?
>  
> Thanks a lot,
> Yuyun


From yyang85 at stanford.edu  Tue Apr 16 09:29:36 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Tue, 16 Apr 2019 14:29:36 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
Message-ID: <DM6PR02MB524218828CAD251F331DBC52D2240@DM6PR02MB5242.namprd02.prod.outlook.com>

Great, thank you for the advice!

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Tuesday, April 16, 2019 5:54:15 AM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak


  Please try the flag -options_dump this tries to give a much more concise view of what objects have not been freed. For example I commented
out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:

./ex19 -objects_dump
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
The following objects were never freed
-----------------------------------------
[0] DM da DM_0x84000000_0
      [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
[0] Vec seq Vec_0x84000000_6
      [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c

Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free.

Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
For example because the unfreed vector holds a reference to the DM the DM is listed as not freed. Once you properly destroy the vector you'll
not that the DM is no longer listed as non freed.

It can be a little overwhelming at first to figure out what objects have not been freed. We recommending setting the environmental variable
export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from
the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.

   Good luck



   Barry


> On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Hello team,
>
> I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
>
> [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
>       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
>       [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c
>
> The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
>
> How can I use this option to debug effectively?
>
> Thanks a lot,
> Yuyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/f4c16d60/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 09:32:40 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 09:32:40 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
Message-ID: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>

Hi PETSc developers,

I?m trying to write a large vector created with VecCreateMPI (size
32768x32768) concurrently from 4 nodes (+32 tasks per node, total 128
mpi-ranks) and I see the following (indicative) error : [Full error log is
here : https://file.io/CdjUfe]

HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 52:
  #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #001: H5Dint.c line 329 in H5D__create_named(): unable to create and
link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: H5L.c line 1557 in H5L_link_object(): unable to create new
link to object
    major: Links
    minor: Unable to initialize object
  #003: H5L.c line 1798 in H5L__create_real(): can't insert link
    major: Links
    minor: Unable to insert object
  #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
    major: Symbol table
HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 59:
  #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #001: H5Dint.c line 329 in H5D__create_named(): unable to create and
link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: H5L.c line 1557 in H5L_link_object(): unable to create new
link to object
    major: Links
    minor: Unable to initialize object
  #003: H5L.c line 1798 in H5L__create_real(): can't insert link
    major: Links
    minor: Unable to insert object
  #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path
traversal failed
    major: Symbol table
    minor: Object not found
  #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal
operator failed
    major: Symbol table
    minor: Callback failed
  #006: H5L.c line 1604 in H5L__link_cb(): unable to create object
    major: Links
    minor: Unable to initialize object
  #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #008: H5Doh.c line 300 in H5O__dset_create(): unable to create
dataset
    minor: Object not found
  #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal
operator failed
    major: Symbol table
    minor: Callback failed
  #006: H5L.c line 1604 in H5L__link_cb(): unable to create object
    major: Links
    minor: Unable to initialize object
  #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #008: H5Doh.c line 300 in H5O__dset_create(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
  #009: H5Dint.c line 1274 in H5D__create(): unable to construct
layout information
    major: Dataset
    minor: Unable to initialize object
  #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
    major: Dataset
    minor: Bad value
  #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
    major: Dataset
    minor: Unable to initialize object
    major: Dataset
    minor: Unable to initialize object
  #009: H5Dint.c line 1274 in H5D__create(): unable to construct
layout information
    major: Dataset
    minor: Unable to initialize object
  #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
    major: Dataset
    minor: Bad value
  #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
    major: Dataset
    minor: Unable to initialize object
.......

I spoke to Barry last evening who said that this is a known error that was
fixed for DMDA vecs but is broken for non-dmda vecs.

Could this be fixed ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/94d67896/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 09:36:17 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 09:36:17 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
Message-ID: <CAOGsD9gSrL8sBRAajph0PkgE8=WttP=W86fS65+=GYqyY1kRFw@mail.gmail.com>

Apologies for the broken link. Attached is the full error log.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/5f71ca58/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slurm-1017838.out
Type: application/octet-stream
Size: 1005787 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/5f71ca58/attachment-0001.obj>

From knepley at gmail.com  Tue Apr 16 09:43:21 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Apr 2019 10:43:21 -0400
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
Message-ID: <CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>

On Tue, Apr 16, 2019 at 10:34 AM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi PETSc developers,
>
> I?m trying to write a large vector created with VecCreateMPI (size
> 32768x32768) concurrently from 4 nodes (+32 tasks per node, total 128
> mpi-ranks) and I see the following (indicative) error : [Full error log is
> here : https://file.io/CdjUfe]
>
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 52:
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object
>     major: Links
>     minor: Unable to initialize object
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link
>     major: Links
>     minor: Unable to insert object
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
>     major: Symbol table
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 59:
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object
>     major: Links
>     minor: Unable to initialize object
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link
>     major: Links
>     minor: Unable to insert object
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
>     major: Symbol table
>     minor: Object not found
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed
>     major: Symbol table
>     minor: Callback failed
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object
>     major: Links
>     minor: Unable to initialize object
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object
>     major: Object header
>     minor: Can't open object
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset
>     minor: Object not found
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed
>     major: Symbol table
>     minor: Callback failed
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object
>     major: Links
>     minor: Unable to initialize object
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object
>     major: Object header
>     minor: Can't open object
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
>     major: Dataset
>     minor: Unable to initialize object
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
> .......
>
> I spoke to Barry last evening who said that this is a known error that was
> fixed for DMDA vecs but is broken for non-dmda vecs.
>
> Could this be fixed ?
>
Barry, what is broken here?


https://bitbucket.org/petsc/petsc/src/8d0323b9f315b69b7b31c5e966afbeb37eb6eb24/src/vec/vec/impls/mpi/pdvec.c#lines-757

Go in the debugger and see what chunksize is being set here, since it
should be less than 4GB, unless HDF5_INT_MAX is somehow wrong
in your installation.

  Thanks,

     Matt

> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/f9e77492/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 09:57:34 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 09:57:34 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
Message-ID: <CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>

Hi Matt,

I tried running the same example with a smaller grid on a workstation and I
see that for a grid size of 8192x8192 (vector write dims 67108864, 2), the
output file has a chunk size of (16777215, 2).

I can?t see HDF5_INT_MAX in the spack build-log (which includes configure).
Is there a better way to look it up?

[sajid at xrmlite .spack]$ cat build.out | grep "HDF"
#define PETSC_HAVE_HDF5 1
#define PETSC_HAVE_LIBHDF5HL_FORTRAN 1
#define PETSC_HAVE_LIBHDF5 1
#define PETSC_HAVE_LIBHDF5_HL 1
#define PETSC_HAVE_LIBHDF5_FORTRAN 1
#define PETSC_HAVE_HDF5_RELEASE_VERSION 5
#define PETSC_HAVE_HDF5_MINOR_VERSION 10
#define PETSC_HAVE_HDF5_MAJOR_VERSION 1

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/86f634da/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 11:15:04 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 11:15:04 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
	<7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
Message-ID: <CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>

Hi Barry/Matt,

Since VecDuplicate calls v->ops->duplicate, can't we just add custom
duplicate ops to the (f_in/f_out/b_out) vectors when they are created via
MatCreateFFTW? (just like the custom destroy ops are defined)

Also, what is the PetscObjectStateIncrease function doing ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/42db9b8b/attachment.html>

From knepley at gmail.com  Tue Apr 16 12:53:13 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Apr 2019 13:53:13 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
	<7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
	<CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>
Message-ID: <CAMYG4GkwvX8WcoR8cSDaP2MP+N2e1fKbHDa=EHMK-gRLCGgLQQ@mail.gmail.com>

On Tue, Apr 16, 2019 at 12:15 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Hi Barry/Matt,
>
> Since VecDuplicate calls v->ops->duplicate, can't we just add custom
> duplicate ops to the (f_in/f_out/b_out) vectors when they are created via
> MatCreateFFTW? (just like the custom destroy ops are defined)
>

You could.


> Also, what is the PetscObjectStateIncrease function doing ?
>

It increments the integer marking the state of that object. We use this to
indicates that the values have been changed.
This allows us to do things like avoid recomputing norms if not values have
changed.

  Thanks,

    Matt


> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/08ada4de/attachment.html>

From knepley at gmail.com  Tue Apr 16 13:13:18 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Apr 2019 14:13:18 -0400
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
Message-ID: <CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>

On Tue, Apr 16, 2019 at 10:58 AM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Hi Matt,
>
> I tried running the same example with a smaller grid on a workstation and
> I see that for a grid size of 8192x8192 (vector write dims 67108864, 2),
> the output file has a chunk size of (16777215, 2).
>
> I want to see what chunksize is when you get the failure.


> I can?t see HDF5_INT_MAX in the spack build-log (which includes
> configure). Is there a better way to look it up?
>
master *:~/TripReceipts/RiceTomDefense2019$ find
/PETSc3/petsc/petsc-dev/include/ -name "*.h" | xargs grep PETSC_HDF5

/PETSc3/petsc/petsc-dev/include//petscviewerhdf5.h:#define
PETSC_HDF5_INT_MAX  2147483647

/PETSc3/petsc/petsc-dev/include//petscviewerhdf5.h:#define
PETSC_HDF5_INT_MIN -2147483647

/PETSc3/petsc/petsc-dev/include//petscviewerhdf5.h:  if ((a) >
PETSC_HDF5_INT_MAX) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE,"Array
too long for HDF5");

  Matt

> [sajid at xrmlite .spack]$ cat build.out | grep "HDF"
> #define PETSC_HAVE_HDF5 1
> #define PETSC_HAVE_LIBHDF5HL_FORTRAN 1
> #define PETSC_HAVE_LIBHDF5 1
> #define PETSC_HAVE_LIBHDF5_HL 1
> #define PETSC_HAVE_LIBHDF5_FORTRAN 1
> #define PETSC_HAVE_HDF5_RELEASE_VERSION 5
> #define PETSC_HAVE_HDF5_MINOR_VERSION 10
> #define PETSC_HAVE_HDF5_MAJOR_VERSION 1
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/aef8b8be/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 14:43:44 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 14:43:44 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
Message-ID: <CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>

So, I tried running the debug version with valgrind to see if I can find
the chunk size that's being set but I don't see it. Is there a better way
to do it ?

`$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out
file is attached.]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/757699c5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out
Type: application/octet-stream
Size: 6010396 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/757699c5/attachment-0001.obj>

From knepley at gmail.com  Tue Apr 16 14:52:06 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Apr 2019 15:52:06 -0400
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
Message-ID: <CAMYG4GmiRqDF75dwz0gdVuFWH3SNSDCocGZTJtqhBSg23appeg@mail.gmail.com>

On Tue, Apr 16, 2019 at 3:44 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> So, I tried running the debug version with valgrind to see if I can find
> the chunk size that's being set but I don't see it. Is there a better way
> to do it ?
>
> `$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out
> file is attached.]
>

valgrind does not do anything here. Just put a breakpoint in the code
section I shared and print chunksize.

  Thanks,

     Matt
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/0f0750cb/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr 16 17:13:40 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 16 Apr 2019 22:13:40 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
Message-ID: <8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>


  So valgrind is printing all kinds of juicy information about uninitialized values but it is all worthless because MPICH was not built by spack to be valgrind clean. We can't know if any of the problems valgrind flags are real. MPICH needs to be configured with the option --enable-g=meminit to be valgrind clean. PETSc's --download-mpich always installs a valgrind clean MPI. 

It is unfortunate Spack doesn't provide a variant of MPICH that is valgrind clean; actually it should default to valgrind clean MPICH.

  Barry




> On Apr 16, 2019, at 2:43 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> So, I tried running the debug version with valgrind to see if I can find the chunk size that's being set but I don't see it. Is there a better way to do it ?
> 
> `$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out file is attached.]
> <out>


From bsmith at mcs.anl.gov  Tue Apr 16 17:15:32 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 16 Apr 2019 22:15:32 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
Message-ID: <CBCCD2DC-B8BC-4C38-A2A6-20D173D0AB91@anl.gov>


   You can run with -start_in_debugger  -debugger_nodes 0 and in that one debugger window put a break point in the needed vector routine to print the chunk sizes at the appropriate line. 

  Barry


> On Apr 16, 2019, at 9:32 AM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi PETSc developers,
> 
> I?m trying to write a large vector created with VecCreateMPI (size 32768x32768) concurrently from 4 nodes (+32 tasks per node, total 128 mpi-ranks) and I see the following (indicative) error : [Full error log is here : https://file.io/CdjUfe] 
> 
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 52:
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object
>     major: Links
>     minor: Unable to initialize object
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link
>     major: Links
>     minor: Unable to insert object
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
>     major: Symbol table
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 59:                              
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset                        
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset  
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object       
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link                        
>     major: Links                                                                        
>     minor: Unable to insert object                                                      
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed        
>     major: Symbol table                                                                 
>     minor: Object not found                                                             
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed       
>     major: Symbol table                                                                 
>     minor: Callback failed                                                              
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object                      
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object                   
>     major: Object header                                                                
>     minor: Can't open object                                                            
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset                
>     minor: Object not found                                                             
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed       
>     major: Symbol table                                                                 
>     minor: Callback failed                                                              
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object                      
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object                   
>     major: Object header                                                                
>     minor: Can't open object                                                            
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset                
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
>     major: Dataset
>     minor: Unable to initialize object
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
> .......
> 
> I spoke to Barry last evening who said that this is a known error that was fixed for DMDA vecs but is broken for non-dmda vecs.
> 
> Could this be fixed ?
> 
> 
> Thank You, 
> Sajid Ali
> Applied Physics
> Northwestern University


From bsmith at mcs.anl.gov  Tue Apr 16 20:42:49 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 01:42:49 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
Message-ID: <4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>


  Dang, I  ranted too soon. 

  I built mpich  using spack (master branch) and a very old Gnu C compiler and it produced valgrind clean code. Spack definitely is not passing the --enable-g=meminit to MPICH ./configure so this version of MPICH valgrind must be clean by default? MPICH's ./configure has 

meminit  - Preinitialize memory associated structures and unions to
                   eliminate access warnings from programs like valgrind

The default for enable-g is most and 

most|yes)
        perform_memtracing=yes
        enable_append_g=yes
        perform_meminit=yes
        perform_dbgmutex=yes
        perform_mutexnesting=yes
        perform_handlealloc=yes
        perform_handle=yes

So it appears that at least some releases of MPICH are suppose to be valgrind clean by default ;).

Looking back at Sajid's valgrind output more carefully 

Conditional jump or move depends on uninitialised value(s)
==15359==    at 0x1331069A: __intel_sse4_strncmp (in /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64_lin/libintlc.so.5)

is the only valgrind error. Which I remember seeing from using Intel compilers for a long time, nothing to do with MPICH

Thus I conclude that Sajid's code is actually valgrind clean; and I withdraw my rant about MPICH/spack

Barry



> On Apr 16, 2019, at 5:13 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  So valgrind is printing all kinds of juicy information about uninitialized values but it is all worthless because MPICH was not built by spack to be valgrind clean. We can't know if any of the problems valgrind flags are real. MPICH needs to be configured with the option --enable-g=meminit to be valgrind clean. PETSc's --download-mpich always installs a valgrind clean MPI. 
> 
> It is unfortunate Spack doesn't provide a variant of MPICH that is valgrind clean; actually it should default to valgrind clean MPICH.
> 
>  Barry
> 
> 
> 
> 
>> On Apr 16, 2019, at 2:43 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> So, I tried running the debug version with valgrind to see if I can find the chunk size that's being set but I don't see it. Is there a better way to do it ?
>> 
>> `$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out file is attached.]
>> <out>
> 


From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 21:47:22 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 21:47:22 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
Message-ID: <CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>

Quick question : To drop a print statement at the required location, I need
to modify the source code, build petsc from source and compile with this
new version of petsc, right or is there an easier way? (Just to confirm
before putting in the effort)

On Tue, Apr 16, 2019 at 8:42 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   Dang, I  ranted too soon.
>
>   I built mpich  using spack (master branch) and a very old Gnu C compiler
> and it produced valgrind clean code. Spack definitely is not passing the
> --enable-g=meminit to MPICH ./configure so this version of MPICH valgrind
> must be clean by default? MPICH's ./configure has
>
> meminit  - Preinitialize memory associated structures and unions to
>                    eliminate access warnings from programs like valgrind
>
> The default for enable-g is most and
>
> most|yes)
>         perform_memtracing=yes
>         enable_append_g=yes
>         perform_meminit=yes
>         perform_dbgmutex=yes
>         perform_mutexnesting=yes
>         perform_handlealloc=yes
>         perform_handle=yes
>
> So it appears that at least some releases of MPICH are suppose to be
> valgrind clean by default ;).
>
> Looking back at Sajid's valgrind output more carefully
>
> Conditional jump or move depends on uninitialised value(s)
> ==15359==    at 0x1331069A: __intel_sse4_strncmp (in
> /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64_lin/libintlc.so.5)
>
> is the only valgrind error. Which I remember seeing from using Intel
> compilers for a long time, nothing to do with MPICH
>
> Thus I conclude that Sajid's code is actually valgrind clean; and I
> withdraw my rant about MPICH/spack
>
> Barry
>
>
>
> > On Apr 16, 2019, at 5:13 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> >
> >
> >  So valgrind is printing all kinds of juicy information about
> uninitialized values but it is all worthless because MPICH was not built by
> spack to be valgrind clean. We can't know if any of the problems valgrind
> flags are real. MPICH needs to be configured with the option
> --enable-g=meminit to be valgrind clean. PETSc's --download-mpich always
> installs a valgrind clean MPI.
> >
> > It is unfortunate Spack doesn't provide a variant of MPICH that is
> valgrind clean; actually it should default to valgrind clean MPICH.
> >
> >  Barry
> >
> >
> >
> >
> >> On Apr 16, 2019, at 2:43 PM, Sajid Ali via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>
> >> So, I tried running the debug version with valgrind to see if I can
> find the chunk size that's being set but I don't see it. Is there a better
> way to do it ?
> >>
> >> `$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out
> file is attached.]
> >> <out>
> >
>
>

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/62f565cc/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Apr 16 21:52:45 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 02:52:45 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
Message-ID: <757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>


  Funny you should ask, I just found the bug. 

> On Apr 16, 2019, at 9:47 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> Quick question : To drop a print statement at the required location, I need to modify the source code, build petsc from source and compile with this new version of petsc, right or is there an easier way? (Just to confirm before putting in the effort)

   Yes. But perhaps spack has a way to handle this as well; it should. Satish? If you can get spack to use the git repository then you could edit in that and somehow have spack rebuild using your edited repository.

   Barry
> 
> On Tue, Apr 16, 2019 at 8:42 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   Dang, I  ranted too soon.
> 
>   I built mpich  using spack (master branch) and a very old Gnu C compiler and it produced valgrind clean code. Spack definitely is not passing the --enable-g=meminit to MPICH ./configure so this version of MPICH valgrind must be clean by default? MPICH's ./configure has
> 
> meminit  - Preinitialize memory associated structures and unions to
>                    eliminate access warnings from programs like valgrind
> 
> The default for enable-g is most and
> 
> most|yes)
>         perform_memtracing=yes
>         enable_append_g=yes
>         perform_meminit=yes
>         perform_dbgmutex=yes
>         perform_mutexnesting=yes
>         perform_handlealloc=yes
>         perform_handle=yes
> 
> So it appears that at least some releases of MPICH are suppose to be valgrind clean by default ;).
> 
> Looking back at Sajid's valgrind output more carefully
> 
> Conditional jump or move depends on uninitialised value(s)
> ==15359==    at 0x1331069A: __intel_sse4_strncmp (in /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64_lin/libintlc.so.5)
> 
> is the only valgrind error. Which I remember seeing from using Intel compilers for a long time, nothing to do with MPICH
> 
> Thus I conclude that Sajid's code is actually valgrind clean; and I withdraw my rant about MPICH/spack
> 
> Barry
> 
> 
> 
> > On Apr 16, 2019, at 5:13 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> >
> >
> >  So valgrind is printing all kinds of juicy information about uninitialized values but it is all worthless because MPICH was not built by spack to be valgrind clean. We can't know if any of the problems valgrind flags are real. MPICH needs to be configured with the option --enable-g=meminit to be valgrind clean. PETSc's --download-mpich always installs a valgrind clean MPI.
> >
> > It is unfortunate Spack doesn't provide a variant of MPICH that is valgrind clean; actually it should default to valgrind clean MPICH.
> >
> >  Barry
> >
> >
> >
> >
> >> On Apr 16, 2019, at 2:43 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> >>
> >> So, I tried running the debug version with valgrind to see if I can find the chunk size that's being set but I don't see it. Is there a better way to do it ?
> >>
> >> `$ mpirun -np 32 valgrind ./ex_ms -prop_steps 1 -info &> out`. [The out file is attached.]
> >> <out>
> >
> 
> 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University


From bsmith at mcs.anl.gov  Tue Apr 16 22:17:05 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 03:17:05 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
Message-ID: <6363155F-7526-41D8-9591-CDC16D31B0C6@anl.gov>

https://bitbucket.org/petsc/petsc/pull-requests/1551/chunksize-could-overflow-and-become/diff

With this fix I can run with your vector size on 1 process. With 2 processes I get

$ petscmpiexec -n 2 ./ex1 
Assertion failed in file adio/common/ad_write_coll.c at line 904: (curr_to_proc[p] + len - done_to_proc[p]) == (unsigned) (curr_to_proc[p] + len - done_to_proc[p])
0   libpmpi.0.dylib                     0x0000000111241f3e backtrace_libc + 62
1   libpmpi.0.dylib                     0x0000000111241ef5 MPL_backtrace_show + 21
2   libpmpi.0.dylib                     0x000000011119f85a MPIR_Assert_fail + 90
3   libpmpi.0.dylib                     0x00000001111a15f3 MPIR_Ext_assert_fail + 35
4   libmpi.0.dylib                      0x0000000110eee16e ADIOI_Fill_send_buffer + 1134
5   libmpi.0.dylib                      0x0000000110eefe74 ADIOI_W_Exchange_data + 2980
6   libmpi.0.dylib                      0x0000000110eed7ad ADIOI_Exch_and_write + 3197
7   libmpi.0.dylib                      0x0000000110eec854 ADIOI_GEN_WriteStridedColl + 2004
8   libpmpi.0.dylib                     0x000000011128ad4b MPIOI_File_write_all + 1179
9   libmpi.0.dylib                      0x0000000110ec382b MPI_File_write_at_all + 91
10  libhdf5.10.dylib                    0x00000001108b982a H5FD_mpio_write + 1466
11  libhdf5.10.dylib                    0x00000001108b127a H5FD_write + 634
12  li

Looks like an int overflow in the MPIIO. (It is scary to see the ints in the ADIO code as opposed to 64 bit integers but I guess somehow it works, maybe this is a strange corner case and I don't know if the problem is with HDF5 or MPIIO) 

 on 4 and 8 processes it runs. 

Note that you are playing with a very dangerous size. 32768 * 32768 * 2 is a negative number in int. So this is essentially the largest problem you can run before switching to 64 bit indices for PETSc. 

  Barry



> On Apr 16, 2019, at 9:32 AM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi PETSc developers,
> 
> I?m trying to write a large vector created with VecCreateMPI (size 32768x32768) concurrently from 4 nodes (+32 tasks per node, total 128 mpi-ranks) and I see the following (indicative) error : [Full error log is here : https://file.io/CdjUfe] 
> 
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 52:
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset
>     major: Dataset
>     minor: Unable to initialize object
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object
>     major: Links
>     minor: Unable to initialize object
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link
>     major: Links
>     minor: Unable to insert object
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
>     major: Symbol table
> HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 59:                              
>   #000: H5D.c line 145 in H5Dcreate2(): unable to create dataset                        
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #001: H5Dint.c line 329 in H5D__create_named(): unable to create and link to dataset  
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #002: H5L.c line 1557 in H5L_link_object(): unable to create new link to object       
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #003: H5L.c line 1798 in H5L__create_real(): can't insert link                        
>     major: Links                                                                        
>     minor: Unable to insert object                                                      
>   #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed        
>     major: Symbol table                                                                 
>     minor: Object not found                                                             
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed       
>     major: Symbol table                                                                 
>     minor: Callback failed                                                              
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object                      
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object                   
>     major: Object header                                                                
>     minor: Can't open object                                                            
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset                
>     minor: Object not found                                                             
>   #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed       
>     major: Symbol table                                                                 
>     minor: Callback failed                                                              
>   #006: H5L.c line 1604 in H5L__link_cb(): unable to create object                      
>     major: Links                                                                        
>     minor: Unable to initialize object                                                  
>   #007: H5Oint.c line 2453 in H5O_obj_create(): unable to open object                   
>     major: Object header                                                                
>     minor: Can't open object                                                            
>   #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset                
>     major: Dataset                                                                      
>     minor: Unable to initialize object                                                  
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
>     major: Dataset
>     minor: Unable to initialize object
>   #009: H5Dint.c line 1274 in H5D__create(): unable to construct layout information
>     major: Dataset
>     minor: Unable to initialize object
>   #010: H5Dchunk.c line 872 in H5D__chunk_construct(): unable to set chunk sizes
>     major: Dataset
>     minor: Bad value
>   #011: H5Dchunk.c line 831 in H5D__chunk_set_sizes(): chunk size must be < 4GB
>     major: Dataset
>     minor: Unable to initialize object
> .......
> 
> I spoke to Barry last evening who said that this is a known error that was fixed for DMDA vecs but is broken for non-dmda vecs.
> 
> Could this be fixed ?
> 
> 
> Thank You, 
> Sajid Ali
> Applied Physics
> Northwestern University


From balay at mcs.anl.gov  Tue Apr 16 22:23:50 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 03:23:50 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
Message-ID: <alpine.LFD.2.21.1904162212000.2713@sb>

On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:

> 
>   Funny you should ask, I just found the bug. 
> 
> > On Apr 16, 2019, at 9:47 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> > 
> > Quick question : To drop a print statement at the required location, I need to modify the source code, build petsc from source and compile with this new version of petsc, right or is there an easier way? (Just to confirm before putting in the effort)
> 
>    Yes. But perhaps spack has a way to handle this as well; it should. Satish? If you can get spack to use the git repository then you could edit in that and somehow have spack rebuild using your edited repository.


$ spack help install |grep stage
  --keep-stage          don't remove the build stage if installation succeeds
  --dont-restage        if a partial install is detected, don't delete prior state

Here is how it works.

- By default - spack downloads the tarball/git-snapshots and saves them in var/spack/cache
- and it stages them for build in var/spack/stage [i.e untar and ready to compile]
- after the build is complete - it installs in opt/.. and deletes the staged/build files.
 [if the build breaks - it leaves the stage alone]

So if we want to add some modifications to a broken build and rebuild - I would:

- 'spack stage' or 'spack install --keep-stage' [to get the package files staged but not deleted]
- edit files in stage
- 'spack install --dont-restage --keep-stage'
  i.e use the currently staged files and build from it. And don't delete them even if the build succeeds

Satish

From bsmith at mcs.anl.gov  Tue Apr 16 22:30:46 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 03:30:46 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904162212000.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
Message-ID: <69CFF03D-209F-430E-B904-1BFFFB8A8161@mcs.anl.gov>


  Satish,

    Thanks for the instructions. Since eventually we want everyone to use spack (but with the git repository of PETSc) is there a place we should 
document this? Perhaps at the bottom of the installation.html?

  Barry

Note: of course we want them to be able to painlessly make pull requests from the thing spack downloads. 


> On Apr 16, 2019, at 10:23 PM, Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:
> 
>> 
>>  Funny you should ask, I just found the bug. 
>> 
>>> On Apr 16, 2019, at 9:47 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
>>> 
>>> Quick question : To drop a print statement at the required location, I need to modify the source code, build petsc from source and compile with this new version of petsc, right or is there an easier way? (Just to confirm before putting in the effort)
>> 
>>   Yes. But perhaps spack has a way to handle this as well; it should. Satish? If you can get spack to use the git repository then you could edit in that and somehow have spack rebuild using your edited repository.
> 
> 
> $ spack help install |grep stage
>  --keep-stage          don't remove the build stage if installation succeeds
>  --dont-restage        if a partial install is detected, don't delete prior state
> 
> Here is how it works.
> 
> - By default - spack downloads the tarball/git-snapshots and saves them in var/spack/cache
> - and it stages them for build in var/spack/stage [i.e untar and ready to compile]
> - after the build is complete - it installs in opt/.. and deletes the staged/build files.
> [if the build breaks - it leaves the stage alone]
> 
> So if we want to add some modifications to a broken build and rebuild - I would:
> 
> - 'spack stage' or 'spack install --keep-stage' [to get the package files staged but not deleted]
> - edit files in stage
> - 'spack install --dont-restage --keep-stage'
>  i.e use the currently staged files and build from it. And don't delete them even if the build succeeds
> 
> Satish


From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 22:30:54 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 22:30:54 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904162212000.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
Message-ID: <CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>

@Barry: Thanks for the bugfix!

@Satish: Thanks for pointing out this method!

My preferred way previously was to download the source code, unzip, edit,
zip. Now ask spack to not checksum (because my edit has changed stuff) and
build. Lately, spack has added git support and now I create a branch of
spack where I add my bugfix branch as the default build git repo instead of
master to now deal with checksum headaches.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/3e52306f/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr 16 22:38:19 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 03:38:19 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
Message-ID: <76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>



> On Apr 16, 2019, at 10:30 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> @Barry: Thanks for the bugfix! 
> 
> @Satish: Thanks for pointing out this method!
> 
> My preferred way previously was to download the source code, unzip, edit, zip. Now ask spack to not checksum (because my edit has changed stuff) and build. Lately, spack has added git support and now I create a branch of spack where I add my bugfix branch as the default build git repo instead of master to now deal with checksum headaches. 

   With the PETSc build system directly it handles dependencies, that is if you use a PETSC_ARCH and edit one PETSc file it will only recompile that one file and add it to the library instead of insisting on recompiling all of PETSc (as developers of course we rely on this or we'd go insane waiting for builds to complete when we are adding code).

  Satish,

     Is this possible with spack?

  Barry


> 


From balay at mcs.anl.gov  Tue Apr 16 22:58:58 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 03:58:58 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904162243200.2713@sb>

On Tue, 16 Apr 2019, Sajid Ali wrote:

> Lately, spack has added git support and now I create a branch of
> spack where I add my bugfix branch as the default build git repo instead of
> master to now deal with checksum headaches.

Some good and bad here..

    version('develop', branch='master')
    version('3.11.99', branch='maint')
    version('maint', branch='maint')

git branch is the only way to get a rebuild to pick up package changes (in branch)

However its best to set appropriate version numbers here. i.e
'3.11.99' should be preferable over 'maint'. Otherwise spack version
comparison logic will give unwanted results. It does stuff like:

develop > 3.11.99 > 3.10.xx > maint (or other strings)

Wrt tarballs and commit-ids - spack saves them as tarballs in cache
and reuses them. For ex: - the download below will be saved as
petsc-3.10.1.tar.gz.  Even if you change commit from 'abc' to 'def'
spack won't recognize this change and use the cached tarball.

However - the bad part wrt branch is - each time you do a 'spack
install' - it does a git clone.  [i.e there is no local git clone
which does a 'fetch' to minimize the clone overhead]

Satish

From balay at mcs.anl.gov  Tue Apr 16 23:04:06 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 04:04:06 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
Message-ID: <alpine.LFD.2.21.1904162259040.2713@sb>

On Tue, 16 Apr 2019, Smith, Barry F. wrote:

> 
> 
> > On Apr 16, 2019, at 10:30 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> > 
> > @Barry: Thanks for the bugfix! 
> > 
> > @Satish: Thanks for pointing out this method!
> > 
> > My preferred way previously was to download the source code, unzip, edit, zip. Now ask spack to not checksum (because my edit has changed stuff) and build. Lately, spack has added git support and now I create a branch of spack where I add my bugfix branch as the default build git repo instead of master to now deal with checksum headaches. 
> 
>    With the PETSc build system directly it handles dependencies, that is if you use a PETSC_ARCH and edit one PETSc file it will only recompile that one file and add it to the library instead of insisting on recompiling all of PETSc (as developers of course we rely on this or we'd go insane waiting for builds to complete when we are adding code).

Yeah but this is within a single package - and only if we don't redo a configure.

And some of our code to avoid rebuilding external packages have corner cases - so we have to occasionally ask users to do 'rm -rf PETSC_ARCH'

> 
>      Is this possible with spack?

Spack tries to do this [avoid rebuilds] at a package level.

However within a package - it doesn't keep build files. [and if the
user forces spack to not delete them with '--dont-restage
--keep-stage' - it doesn't check if the package need to run configure
again or not etc..] I'm not sure if this is possible to do
consistently without error cases across the package collection spack
has.

Satish

From balay at mcs.anl.gov  Tue Apr 16 23:11:34 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 04:11:34 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904162259040.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
Message-ID: <alpine.LFD.2.21.1904162305550.2713@sb>

On Wed, 17 Apr 2019, Balay, Satish via petsc-users wrote:

> On Tue, 16 Apr 2019, Smith, Barry F. wrote:
> 
> > 
> > 
> > > On Apr 16, 2019, at 10:30 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> > > 
> > > @Barry: Thanks for the bugfix! 
> > > 
> > > @Satish: Thanks for pointing out this method!
> > > 
> > > My preferred way previously was to download the source code, unzip, edit, zip. Now ask spack to not checksum (because my edit has changed stuff) and build. Lately, spack has added git support and now I create a branch of spack where I add my bugfix branch as the default build git repo instead of master to now deal with checksum headaches. 
> > 
> >    With the PETSc build system directly it handles dependencies, that is if you use a PETSC_ARCH and edit one PETSc file it will only recompile that one file and add it to the library instead of insisting on recompiling all of PETSc (as developers of course we rely on this or we'd go insane waiting for builds to complete when we are adding code).
> 
> Yeah but this is within a single package - and only if we don't redo a configure.
> 
> And some of our code to avoid rebuilding external packages have corner cases - so we have to occasionally ask users to do 'rm -rf PETSC_ARCH'
> 
> > 
> >      Is this possible with spack?
> 
> Spack tries to do this [avoid rebuilds] at a package level.
> 
> However within a package - it doesn't keep build files. [and if the
> user forces spack to not delete them with '--dont-restage
> --keep-stage' - it doesn't check if the package need to run configure
> again or not etc..] I'm not sure if this is possible to do
> consistently without error cases across the package collection spack
> has.

One additional note: spack has a way to get into a mode where one can
do the builds manually - primarily for debugging [when builds fail]

spack build-env petsc bash
spack cd petsc
make

But I haven't checked on how to replicate all the steps [configure;
make all; make install; delete] exactly as what spack would do in
'spack install'

However - in this 'build-env' mode - one could use incremental build
feature provided by any given package.

Satish

From sajidsyed2021 at u.northwestern.edu  Tue Apr 16 23:40:53 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 16 Apr 2019 23:40:53 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904162305550.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
Message-ID: <CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>

> develop > 3.11.99 > 3.10.xx > maint (or other strings)
Just discovered this issue when trying to build with my fork of spack at [1
<https://github.com/s-sajid-ali/spack/commit/05e499571b428f37b8cd1c7d39013e3dec08e5c8>].


So, ideally each developer has to have their develop point to the branch
they want to build ? That would make communication a little confusing since
spack's develop version is some package's master and now everyone wants a
different develop so as to not let spack apply any patches for string
version sorted lower than lowest numeric version.

>Even if you change commit from 'abc' to 'def'spack won't recognize this
change and use the cached tarball.
True, but since checksum changes and the user has to constantly zip and
unzip, I personally find git cloning easier to deal with so it's just a
matter of preference.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190416/c75932f7/attachment-0001.html>

From balay at mcs.anl.gov  Wed Apr 17 00:00:35 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 05:00:35 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904162348580.2713@sb>

On Tue, 16 Apr 2019, Sajid Ali via petsc-users wrote:

> > develop > 3.11.99 > 3.10.xx > maint (or other strings)
> Just discovered this issue when trying to build with my fork of spack at [1
> <https://github.com/s-sajid-ali/spack/commit/05e499571b428f37b8cd1c7d39013e3dec08e5c8>].
> 
> 
> So, ideally each developer has to have their develop point to the branch
> they want to build ? That would make communication a little confusing since
> spack's develop version is some package's master and now everyone wants a
> different develop so as to not let spack apply any patches for string
> version sorted lower than lowest numeric version.

There is some issue filed [with PR?] regarding this with sorting order
of string versions and numerical versions. This might improve in the
future. But for now 'bugfix-vecduplicate-fftw-vec' will be lower than
version 0.1

Also 'develop' might not be appropriate for all branches.

For ex: - petsc has maint, maint-3.10 etc branches. - so if one is
creating a bugfix for maint - (i.e start a branch off maint) it would
be inappropriate to call it 'develop' - as it will be marked > version
3.11.99 and break some of the version comparisons.

> 
> >Even if you change commit from 'abc' to 'def'spack won't recognize this
> change and use the cached tarball.
> True, but since checksum changes and the user has to constantly zip and
> unzip, I personally find git cloning easier to deal with so it's just a
> matter of preference.
> 

Here you are referring to tarballs - where the sha256sum is listed.

     url      = "http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.5.3.tar.gz"
     version('3.11.1', 'cb627f99f7ce1540ebbbf338189f89a5f1ecf3ab3b5b0e357f9e46c209f1fb23')

However - one can also say:

     git      = "https://bitbucket.org/sajid__ali/petsc.git"
     version('3.11.1', commit='f3d32574624d5351549675da8733a2646265404f')

Here - spack downloads the git snapshot as tarball (saves in tarball
cache as petsc-3.11.1.tar.gz - and reuses it) - and there is no
sha256sum listed here to check. If you change this to some-other
commit (perhaps to test a fix) - spack will use the cached tarball -
and not downloaded the snapshot corresponding to the changed commit.

Satish

From bsmith at mcs.anl.gov  Wed Apr 17 00:39:54 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 05:39:54 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904162348580.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
Message-ID: <3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>


  So it sounds like spack is still mostly a "package manager" where people use "static" packages and don't hack the package's code. This is not unreasonable, no other package manager supports hacking a package's code easily, presumably. The problem is that in the HPC world a "packaged"
code is always incomplete and may need hacking or application of newly generated patches and this is painful with static package managers so people want to use the git repository directly and mange the build themselves which negates the advantages of using a package manager.

   Thanks

    Barry

Perhaps if spack had an easier mechanism to allow the user to "point to" local git clones it could get closer to the best of both worlds. Maybe spack could support a list of local repositories and branches in the yaml file. But yes the issue of rerunning the "./configure" stage still comes up.






> On Apr 17, 2019, at 12:00 AM, Balay, Satish via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Tue, 16 Apr 2019, Sajid Ali via petsc-users wrote:
> 
>>> develop > 3.11.99 > 3.10.xx > maint (or other strings)
>> Just discovered this issue when trying to build with my fork of spack at [1
>> <https://github.com/s-sajid-ali/spack/commit/05e499571b428f37b8cd1c7d39013e3dec08e5c8>].
>> 
>> 
>> So, ideally each developer has to have their develop point to the branch
>> they want to build ? That would make communication a little confusing since
>> spack's develop version is some package's master and now everyone wants a
>> different develop so as to not let spack apply any patches for string
>> version sorted lower than lowest numeric version.
> 
> There is some issue filed [with PR?] regarding this with sorting order
> of string versions and numerical versions. This might improve in the
> future. But for now 'bugfix-vecduplicate-fftw-vec' will be lower than
> version 0.1
> 
> Also 'develop' might not be appropriate for all branches.
> 
> For ex: - petsc has maint, maint-3.10 etc branches. - so if one is
> creating a bugfix for maint - (i.e start a branch off maint) it would
> be inappropriate to call it 'develop' - as it will be marked > version
> 3.11.99 and break some of the version comparisons.
> 
>> 
>>> Even if you change commit from 'abc' to 'def'spack won't recognize this
>> change and use the cached tarball.
>> True, but since checksum changes and the user has to constantly zip and
>> unzip, I personally find git cloning easier to deal with so it's just a
>> matter of preference.
>> 
> 
> Here you are referring to tarballs - where the sha256sum is listed.
> 
>     url      = "http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.5.3.tar.gz"
>     version('3.11.1', 'cb627f99f7ce1540ebbbf338189f89a5f1ecf3ab3b5b0e357f9e46c209f1fb23')
> 
> However - one can also say:
> 
>     git      = "https://bitbucket.org/sajid__ali/petsc.git"
>     version('3.11.1', commit='f3d32574624d5351549675da8733a2646265404f')
> 
> Here - spack downloads the git snapshot as tarball (saves in tarball
> cache as petsc-3.11.1.tar.gz - and reuses it) - and there is no
> sha256sum listed here to check. If you change this to some-other
> commit (perhaps to test a fix) - spack will use the cached tarball -
> and not downloaded the snapshot corresponding to the changed commit.
> 
> Satish


From sajidsyed2021 at u.northwestern.edu  Wed Apr 17 00:47:21 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 17 Apr 2019 00:47:21 -0500
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
Message-ID: <CAOGsD9hZR=mzH74uh0LXJv9=uzw=6n_T+gBWD5xcrw9VDt5XHg@mail.gmail.com>

>Perhaps if spack had an easier mechanism to allow the user to "point to"
local git clones it could get closer to the best of both worlds. Maybe
spack could support a list of local repositories and branches in the yaml
file.

I wonder if a local git clone of petsc can become a "mirror" for petsc
spack package, though this is not the intended use of mirrors. Refer to
https://spack.readthedocs.io/en/latest/mirrors.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/a2c07626/attachment.html>

From balay at mcs.anl.gov  Wed Apr 17 00:53:21 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 05:53:21 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
Message-ID: <alpine.LFD.2.21.1904170044570.2713@sb>

On Wed, 17 Apr 2019, Smith, Barry F. wrote:

> 
>   So it sounds like spack is still mostly a "package manager" where people use "static" packages and don't hack the package's code. This is not unreasonable, no other package manager supports hacking a package's code easily, presumably. The problem is that in the HPC world a "packaged"
> code is always incomplete and may need hacking or application of newly generated patches and this is painful with static package managers so people want to use the git repository directly and mange the build themselves which negates the advantages of using a package manager.
> 
>    Thanks
> 
>     Barry
> 
> Perhaps if spack had an easier mechanism to allow the user to "point to" local git clones it could get closer to the best of both worlds. Maybe spack could support a list of local repositories and branches in the yaml file. But yes the issue of rerunning the "./configure" stage still comes up.

$ spack help --all| grep diy
  diy                   do-it-yourself: build from an existing source directory

I haven't explored this mode though. [but useful for packages that are
not already represented in repo]

This more was in instructions for one of the packages - but then I
couldn't figure out equivalent of 'spack spec' vs 'spack install'
[query and check dependencies before installing] with diy

Its not ideal - but having local changes in our spack clones (change
git url, add appropriate version lines to branches that one is working
on) is possible [for a group working in this mode]. But might not be
ideal for folk who might want to easily do a one off PR - in this
mode.

Satish

From jed at jedbrown.org  Wed Apr 17 00:56:20 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 16 Apr 2019 23:56:20 -0600
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
Message-ID: <87a7gpkwe3.fsf@jedbrown.org>

"Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:

>   So it sounds like spack is still mostly a "package manager" where people use "static" packages and don't hack the package's code. This is not unreasonable, no other package manager supports hacking a package's code easily, presumably. The problem is that in the HPC world a "packaged"
> code is always incomplete and may need hacking or application of newly generated patches and this is painful with static package managers so people want to use the git repository directly and mange the build themselves which negates the advantages of using a package manager.

I don't think people "want" to hack the packages that they don't
contribute to.  Spack provides pretty rapid distribution of patches.

What if PETSc had

  ./configure --with-mumps=spack

or some alternative that would check with spack to find a suitable
MUMPS, installing it (with Spack's dependency resolution) if not
available?  Then you could hack on PETSc with multiple PETSC_ARCH,
branches, incremental rebuilds, and testing, but not need to deal with
PETSc's crude package installation and upgrade mechanism.

Upon completion, the build could offer a yaml snippet for packages.yaml
in case the user wanted other Spack packages to use that PETSc.

From bsmith at mcs.anl.gov  Wed Apr 17 00:59:46 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 05:59:46 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
	<DM6PR02MB524218828CAD251F331DBC52D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <A7195A1D-9AA5-4DF9-8D94-A3AE1D60CDBE@mcs.anl.gov>


  Please remember to respond-to-all in your emails, otherwise the mail comes only to me and I may not have full information to respond to your queries correctly. 



> On Apr 16, 2019, at 11:56 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> So using -objects_dump showed nothing below the line:
> The following objects were never freed
> -----------------------------------------
> Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?

  No idea. If you have messages from -malloc_dump then memory has not been freed and almost always this is due to undestroyed objects. Presumably you didn't turn off logging in ./configure (nobody does that). If you use the option -objects_dump this is processed in PetscOptionsCheckInitial_Private() which is called from PetscInitialize() with 
#if defined(PETSC_USE_LOG)
  ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&PetscObjectsLog);CHKERRQ(ierr);
#endif
then in inherit.c each object creation gets logged with

PetscErrorCode  PetscHeaderCreate_Private(PetscObject h,PetscClassId classid,const char class_name[],const char descr[],const char mansec[],
                                          MPI_Comm comm,PetscObjectDestroyFunction destroy,PetscObjectViewFunction view)
{
  static PetscInt idcnt = 1;
  PetscErrorCode  ierr;
#if defined(PETSC_USE_LOG)
  PetscObject     *newPetscObjects;
  PetscInt         newPetscObjectsMaxCounts,i;
#endif

  PetscFunctionBegin;
  h->classid               = classid;
  h->type                  = 0;
  h->class_name            = (char*)class_name;
  h->description           = (char*)descr;
  h->mansec                = (char*)mansec;
  h->prefix                = 0;
  h->refct                 = 1;
#if defined(PETSC_HAVE_SAWS)
  h->amsmem                = PETSC_FALSE;
#endif
  h->id                    = idcnt++;
  h->parentid              = 0;
  h->qlist                 = 0;
  h->olist                 = 0;
  h->bops->destroy         = destroy;
  h->bops->view            = view;
  h->bops->getcomm         = PetscObjectGetComm_Petsc;
  h->bops->compose         = PetscObjectCompose_Petsc;
  h->bops->query           = PetscObjectQuery_Petsc;
  h->bops->composefunction = PetscObjectComposeFunction_Petsc;
  h->bops->queryfunction   = PetscObjectQueryFunction_Petsc;

  ierr = PetscCommDuplicate(comm,&h->comm,&h->tag);CHKERRQ(ierr);

#if defined(PETSC_USE_LOG)
  /* Keep a record of object created */
  if (PetscObjectsLog) {
    PetscObjectsCounts++;
    for (i=0; i<PetscObjectsMaxCounts; i++) {
      if (!PetscObjects[i]) {
        PetscObjects[i] = h;
        PetscFunctionReturn(0);
      }
    }
    /* Need to increase the space for storing PETSc objects */
    if (!PetscObjectsMaxCounts) newPetscObjectsMaxCounts = 100;
    else                        newPetscObjectsMaxCounts = 2*PetscObjectsMaxCounts;
    ierr = PetscMalloc1(newPetscObjectsMaxCounts,&newPetscObjects);CHKERRQ(ierr);
    ierr = PetscMemcpy(newPetscObjects,PetscObjects,PetscObjectsMaxCounts*sizeof(PetscObject));CHKERRQ(ierr);
    ierr = PetscMemzero(newPetscObjects+PetscObjectsMaxCounts,(newPetscObjectsMaxCounts - PetscObjectsMaxCounts)*sizeof(PetscObject));CHKERRQ(ierr);
    ierr = PetscFree(PetscObjects);CHKERRQ(ierr);

    PetscObjects                        = newPetscObjects;
    PetscObjects[PetscObjectsMaxCounts] = h;
    PetscObjectsMaxCounts               = newPetscObjectsMaxCounts;
  }
#endif
  PetscFunctionReturn(0);
}

Then in PetscFinalize() 

#if defined(PETSC_USE_LOG)
  /*
       List all objects the user may have forgot to free
  */
  if (PetscObjectsLog) {
    ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&flg1);CHKERRQ(ierr);
    if (flg1) {
      MPI_Comm local_comm;
      char     string[64];

      ierr = PetscOptionsGetString(NULL,NULL,"-objects_dump",string,64,NULL);CHKERRQ(ierr);
      ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr);
      ierr = PetscSequentialPhaseBegin_Private(local_comm,1);CHKERRQ(ierr);
      ierr = PetscObjectsDump(stdout,(string[0] == 'a') ? PETSC_TRUE : PETSC_FALSE);CHKERRQ(ierr);
      ierr = PetscSequentialPhaseEnd_Private(local_comm,1);CHKERRQ(ierr);
      ierr = MPI_Comm_free(&local_comm);CHKERRQ(ierr);
    }
  }
#endif

Very simple code. 

Try 

-objects_dump all 

and see if that changes things. Otherwise you'll need to use the debugger to track why it is not printing out the undestroyed objects.


>  
> I also ran valgrind, and am getting the following error when using --track-origins=yes:
> Uninitialised value was created by a stack allocation
> ==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)
>  
> The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?

  Should be unrelated to the -malloc_dump message.

  A "stack allocation" is usually a local variable, while the value from VecGetArray() is from the heap. Check the use of all your local variables in this function.

  Barry


>  
> Thanks for your help on this.
>  
> Best regards,
> Yuyun
>  
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Yuyun Yang via petsc-users
> Sent: Tuesday, April 16, 2019 7:30 AM
> To: Smith, Barry F. <bsmith at mcs.anl.gov>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> Great, thank you for the advice!
>  
> Best regards,
> Yuyun
>  
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, April 16, 2019 5:54:15 AM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> 
>   Please try the flag -options_dump this tries to give a much more concise view of what objects have not been freed. For example I commented
> out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:
> 
> ./ex19 -objects_dump
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> The following objects were never freed
> -----------------------------------------
> [0] DM da DM_0x84000000_0
>       [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
>       [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> [0] Vec seq Vec_0x84000000_6
>       [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
>       [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> 
> Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free. 
> 
> Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
> For example because the unfreed vector holds a reference to the DM the DM is listed as not freed. Once you properly destroy the vector you'll 
> not that the DM is no longer listed as non freed.
> 
> It can be a little overwhelming at first to figure out what objects have not been freed. We recommending setting the environmental variable
> export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from
> the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.
> 
>    Good luck
> 
> 
> 
>    Barry
> 
> 
> > On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello team,
> >  
> > I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
> >  
> > [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
> >       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
> >       [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c
> >  
> > The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
> >  
> > How can I use this option to debug effectively?
> >  
> > Thanks a lot,
> > Yuyun
> 


From bsmith at mcs.anl.gov  Wed Apr 17 01:15:19 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 06:15:19 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <87a7gpkwe3.fsf@jedbrown.org>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
Message-ID: <938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>



> On Apr 17, 2019, at 12:56 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
> "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
> 
>>  So it sounds like spack is still mostly a "package manager" where people use "static" packages and don't hack the package's code. This is not unreasonable, no other package manager supports hacking a package's code easily, presumably. The problem is that in the HPC world a "packaged"
>> code is always incomplete and may need hacking or application of newly generated patches and this is painful with static package managers so people want to use the git repository directly and mange the build themselves which negates the advantages of using a package manager.
> 
> I don't think people "want" to hack the packages that they don't
> contribute to.

   Ok, "want" is not the right word, "need" is perhaps more correct.

>  Spack provides pretty rapid distribution of patches.
> 
> What if PETSc had
> 
>  ./configure --with-mumps=spack
> 
> or some alternative that would check with spack to find a suitable
> MUMPS, installing it (with Spack's dependency resolution) if not
> available?  Then you could hack on PETSc with multiple PETSC_ARCH,
> branches, incremental rebuilds, and testing, but not need to deal with
> PETSc's crude package installation and upgrade mechanism.

  This is fine for "hacking" on PETSc but worthless for any other package. Here is my concern, when someone 
realizes there is a problem with a package they are using through a package manager they think, crud I have to

1) find the git repository for this package
2) git clone the package 
3) figure out how to build the package from source, is it ./configure, cmake, what are the needed arguments,... 
4) wait for the entire thing to build 

then I can go in and investigate the problem and provide and test the fix via a pull request. Heck I'm not going to bother.

Thus a lot of potential contributions of small fixes that everyone in the community would benefit from are lost. This is why, for 
me, an ideal HPC package manager provides a trivial process for providing fixes/improvements to other packages.

For example Sajid could have easily figured out the VecView_MPI_HDF5() bug and provided a fix but just the hassle of 
logistics (not ability to solve the problem) prevented him from providing the bug fix to everyone rapidly. 


Barry



> 
> Upon completion, the build could offer a yaml snippet for packages.yaml
> in case the user wanted other Spack packages to use that PETSc.


From balay at mcs.anl.gov  Wed Apr 17 01:29:28 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 06:29:28 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904170044570.2713@sb>
References: <alpine.LFD.2.21.1904170044570.2713@sb>
Message-ID: <alpine.LFD.2.21.1904170103120.2713@sb>

On Wed, 17 Apr 2019, Satish Balay wrote:

> Its not ideal - but having local changes in our spack clones (change
> git url, add appropriate version lines to branches that one is
> working on) is possible [for a group working in this mode].

[balay at pj03 petsc]$ pwd
/home/balay/petsc
[balay at pj03 petsc]$ git branch
* barry/fix-vecview-mpi-hdf5/maint
  maint
  master

[balay at pj03 spack]$ git diff
diff --git a/var/spack/repos/builtin/packages/petsc/package.py b/var/spack/repos/builtin/packages/petsc/package.py
index 3d9686eb2..57932f93c 100644
--- a/var/spack/repos/builtin/packages/petsc/package.py
+++ b/var/spack/repos/builtin/packages/petsc/package.py
@@ -17,13 +17,16 @@ class Petsc(Package):
 
     homepage = "http://www.mcs.anl.gov/petsc/index.html"
     url      = "http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.5.3.tar.gz"
-    git      = "https://bitbucket.org/petsc/petsc.git"
+    git      = "/home/balay/petsc"
 
     maintainers = ['balay', 'barrysmith', 'jedbrown']
 
     version('develop', branch='master')
     version('xsdk-0.2.0', tag='xsdk-0.2.0')
 
+    version('3.11.99-maint', branch='maint')
+    version('3.11.98-fix-vecview-mpi-hdf5', branch='barry/fix-vecview-mpi-hdf5/maint')
+
     version('3.11.1', 'cb627f99f7ce1540ebbbf338189f89a5f1ecf3ab3b5b0e357f9e46c209f1fb23')
     version('3.11.0', 'b3bed2a9263193c84138052a1b92d47299c3490dd24d1d0bf79fb884e71e678a')
     version('3.10.5', '3a81c8406410e0ffa8a3e9f8efcdf2e683cc40613c9bb5cb378a6498f595803e')
[balay at pj03 spack]$ spack install petsc at develop~hdf5~hypre~superlu-dist~metis
==> mpich at 3.3b1 : externally installed in /home/petsc/soft/mpich-3.3b1
==> mpich at 3.3b1 : already registered in DB
==> openblas is already installed in /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/openblas-0.3.5-xokonl5jhgandxuwswkaabijra337l2d
==> python at 3.7.2 : externally installed in /usr
==> python at 3.7.2 : already registered in DB
==> sowing is already installed in /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/sowing-1.1.25-p1-6wiuwedg5ufoabro3ip4j7ssd43a66je
==> Installing petsc
==> Searching for binary cache of petsc
==> Warning: No Spack mirrors are currently configured
==> No binary for petsc found: installing from source
==> Cloning git repository: /home/balay/petsc on branch master
warning: --depth is ignored in local clones; use file:// instead.
==> No checksum needed when fetching with git
==> Already staged petsc-develop-qcfn35x2lvixtk4u5reczf5y3pjq6a3r in /home/balay/spack/var/spack/stage/petsc-develop-qcfn35x2lvixtk4u5reczf5y3pjq6a3r
==> No patches needed for petsc
==> Building petsc [Package]
==> Executing phase: 'install'
==> [2019-04-17-01:18:20.792426] 'make' '-j16' 'install'
==> Successfully installed petsc
  Fetch: 0.82s.  Build: 2m 22.90s.  Total: 2m 23.71s.
[+] /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/petsc-develop-qcfn35x2lvixtk4u5reczf5y3pjq6a3r
[balay at pj03 spack]$ spack install petsc at 3.11.98-fix-vecview-mpi-hdf5~hdf5~hypre~superlu-dist~metis
==> mpich at 3.3b1 : externally installed in /home/petsc/soft/mpich-3.3b1
==> mpich at 3.3b1 : already registered in DB
==> openblas is already installed in /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/openblas-0.3.5-xokonl5jhgandxuwswkaabijra337l2d
==> python at 3.7.2 : externally installed in /usr
==> python at 3.7.2 : already registered in DB
==> Installing petsc
==> Searching for binary cache of petsc
==> Warning: No Spack mirrors are currently configured
==> No binary for petsc found: installing from source
==> Cloning git repository: /home/balay/petsc on branch barry/fix-vecview-mpi-hdf5/maint
warning: --depth is ignored in local clones; use file:// instead.
==> No checksum needed when fetching with git
==> Already staged petsc-3.11.98-fix-vecview-mpi-hdf5-2i6pmomvyacyefcq7hxqe22gogepf6os in /home/balay/spack/var/spack/stage/petsc-3.11.98-fix-vecview-mpi-hdf5-2i6pmomvyacyefcq7hxqe22gogepf6os
==> No patches needed for petsc
==> Building petsc [Package]
==> Executing phase: 'install'
==> Successfully installed petsc
  Fetch: 0.79s.  Build: 2m 31.08s.  Total: 2m 31.88s.
[+] /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/petsc-3.11.98-fix-vecview-mpi-hdf5-2i6pmomvyacyefcq7hxqe22gogepf6os
[balay at pj03 spack]$ spack install petsc at 3.11.99-maint~hdf5~hypre~superlu-dist~metis
==> mpich at 3.3b1 : externally installed in /home/petsc/soft/mpich-3.3b1
==> mpich at 3.3b1 : already registered in DB
==> openblas is already installed in /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/openblas-0.3.5-xokonl5jhgandxuwswkaabijra337l2d
==> python at 3.7.2 : externally installed in /usr
==> python at 3.7.2 : already registered in DB
==> Installing petsc
==> Searching for binary cache of petsc
==> Warning: No Spack mirrors are currently configured
==> No binary for petsc found: installing from source
==> Cloning git repository: /home/balay/petsc on branch maint
warning: --depth is ignored in local clones; use file:// instead.
==> No checksum needed when fetching with git
==> Already staged petsc-3.11.99-maint-ivdk3dhyirqwjv7fplbkvha6d27hhzmk in /home/balay/spack/var/spack/stage/petsc-3.11.99-maint-ivdk3dhyirqwjv7fplbkvha6d27hhzmk
==> No patches needed for petsc
==> Building petsc [Package]
==> Executing phase: 'install'
==> Successfully installed petsc
  Fetch: 0.82s.  Build: 2m 30.67s.  Total: 2m 31.49s.
[+] /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.3.1/petsc-3.11.99-maint-ivdk3dhyirqwjv7fplbkvha6d27hhzmk
[balay at pj03 spack]$ spack find
==> 7 installed packages
-- linux-fedora29-x86_64 / gcc at 8.3.1 ----------------------------
mpich at 3.3b1  openblas at 0.3.5  petsc at 3.11.98-fix-vecview-mpi-hdf5  petsc at 3.11.99-maint  petsc at develop  python at 3.7.2  sowing at 1.1.25-p1
[balay at pj03 spack]$ 

Satish

From balay at mcs.anl.gov  Wed Apr 17 01:35:31 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 06:35:31 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
	<938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
Message-ID: <alpine.LFD.2.21.1904170129360.2713@sb>

On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:

>   This is fine for "hacking" on PETSc but worthless for any other package. Here is my concern, when someone 
> realizes there is a problem with a package they are using through a package manager they think, crud I have to
> 
> 1) find the git repository for this package
> 2) git clone the package 
> 3) figure out how to build the package from source, is it ./configure, cmake, what are the needed arguments,... 
> 4) wait for the entire thing to build 
> 
> then I can go in and investigate the problem and provide and test the fix via a pull request. Heck I'm not going to bother.
> 
> Thus a lot of potential contributions of small fixes that everyone in the community would benefit from are lost. This is why, for 
> me, an ideal HPC package manager provides a trivial process for providing fixes/improvements to other packages.
> 
> For example Sajid could have easily figured out the VecView_MPI_HDF5() bug and provided a fix but just the hassle of 
> logistics (not ability to solve the problem) prevented him from providing the bug fix to everyone rapidly. 


Even without spack and multiple packages - this is not a easy thing to
do. For ex: most of our users install petsc from tarball.

And if they find a bug - they have to go through similar complicated
process [create a bitbucket account, get a fork - learn the petsc PR
process - make a PR etc].

With spack - I stick to the usual process - and don't get bogged down
by 'spack' support for this process.

If I see a breakage - I do 'spack build-env package [this has its own
issues] - attempt a fix - get it first working with a spack build.

[Alternative is to just edit the package file to get my fix - if its a patch I can find]


Once I have it working [the major issue is taken care off]. Then I
have a diff/patch and then worry about how to submit this diff/patch
to upstream.

Sure its a multi step model - and has many trip points. But is not
that our current petsc only model doesn't have any.

Satish

From bsmith at mcs.anl.gov  Wed Apr 17 01:40:13 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 06:40:13 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <alpine.LFD.2.21.1904170129360.2713@sb>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
	<938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
	<alpine.LFD.2.21.1904170129360.2713@sb>
Message-ID: <BCE816D1-F76A-416D-BAEB-0926A2A01FC8@mcs.anl.gov>



> On Apr 17, 2019, at 1:35 AM, Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:
> 
>>  This is fine for "hacking" on PETSc but worthless for any other package. Here is my concern, when someone 
>> realizes there is a problem with a package they are using through a package manager they think, crud I have to
>> 
>> 1) find the git repository for this package
>> 2) git clone the package 
>> 3) figure out how to build the package from source, is it ./configure, cmake, what are the needed arguments,... 
>> 4) wait for the entire thing to build 
>> 
>> then I can go in and investigate the problem and provide and test the fix via a pull request. Heck I'm not going to bother.
>> 
>> Thus a lot of potential contributions of small fixes that everyone in the community would benefit from are lost. This is why, for 
>> me, an ideal HPC package manager provides a trivial process for providing fixes/improvements to other packages.
>> 
>> For example Sajid could have easily figured out the VecView_MPI_HDF5() bug and provided a fix but just the hassle of 
>> logistics (not ability to solve the problem) prevented him from providing the bug fix to everyone rapidly. 
> 

   I never said that any current practices are better than using spack! It is just that perhaps 
with a few tweaks spack could provide a way to fundamentally improve our current practices (which are, as you acknowledge cumbersome).

   Barry


> Even without spack and multiple packages - this is not a easy thing to
> do. For ex: most of our users install petsc from tarball.
> 
> And if they find a bug - they have to go through similar complicated
> process [create a bitbucket account, get a fork - learn the petsc PR
> process - make a PR etc].
> 
> With spack - I stick to the usual process - and don't get bogged down
> by 'spack' support for this process.
> 
> If I see a breakage - I do 'spack build-env package [this has its own
> issues] - attempt a fix - get it first working with a spack build.
> 
> [Alternative is to just edit the package file to get my fix - if its a patch I can find]
> 
> 
> Once I have it working [the major issue is taken care off]. Then I
> have a diff/patch and then worry about how to submit this diff/patch
> to upstream.
> 
> Sure its a multi step model - and has many trip points. But is not
> that our current petsc only model doesn't have any.
> 
> Satish


From knepley at gmail.com  Wed Apr 17 06:49:55 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 17 Apr 2019 07:49:55 -0400
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <BCE816D1-F76A-416D-BAEB-0926A2A01FC8@mcs.anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
	<938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
	<alpine.LFD.2.21.1904170129360.2713@sb>
	<BCE816D1-F76A-416D-BAEB-0926A2A01FC8@mcs.anl.gov>
Message-ID: <CAMYG4Gkw0cGineGHtohkE0OHzL4_hvDYjgYo-tA7JppZv_JoWw@mail.gmail.com>

On Wed, Apr 17, 2019 at 2:40 AM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> > On Apr 17, 2019, at 1:35 AM, Balay, Satish <balay at mcs.anl.gov> wrote:
> >
> > On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:
> >
> >>  This is fine for "hacking" on PETSc but worthless for any other
> package. Here is my concern, when someone
> >> realizes there is a problem with a package they are using through a
> package manager they think, crud I have to
> >>
> >> 1) find the git repository for this package
> >> 2) git clone the package
> >> 3) figure out how to build the package from source, is it ./configure,
> cmake, what are the needed arguments,...
> >> 4) wait for the entire thing to build
> >>
> >> then I can go in and investigate the problem and provide and test the
> fix via a pull request. Heck I'm not going to bother.
> >>
> >> Thus a lot of potential contributions of small fixes that everyone in
> the community would benefit from are lost. This is why, for
> >> me, an ideal HPC package manager provides a trivial process for
> providing fixes/improvements to other packages.
> >>
> >> For example Sajid could have easily figured out the VecView_MPI_HDF5()
> bug and provided a fix but just the hassle of
> >> logistics (not ability to solve the problem) prevented him from
> providing the bug fix to everyone rapidly.
> >
>
>    I never said that any current practices are better than using spack! It
> is just that perhaps
> with a few tweaks spack could provide a way to fundamentally improve our
> current practices (which are, as you acknowledge cumbersome).
>

This sounds like a hopeless pipedream. But maybe.

   Matt


>    Barry
>
> > Even without spack and multiple packages - this is not a easy thing to
> > do. For ex: most of our users install petsc from tarball.
> >
> > And if they find a bug - they have to go through similar complicated
> > process [create a bitbucket account, get a fork - learn the petsc PR
> > process - make a PR etc].
> >
> > With spack - I stick to the usual process - and don't get bogged down
> > by 'spack' support for this process.
> >
> > If I see a breakage - I do 'spack build-env package [this has its own
> > issues] - attempt a fix - get it first working with a spack build.
> >
> > [Alternative is to just edit the package file to get my fix - if its a
> patch I can find]
> >
> >
> > Once I have it working [the major issue is taken care off]. Then I
> > have a diff/patch and then worry about how to submit this diff/patch
> > to upstream.
> >
> > Sure its a multi step model - and has many trip points. But is not
> > that our current petsc only model doesn't have any.
> >
> > Satish
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/fe641546/attachment-0001.html>

From jed at jedbrown.org  Wed Apr 17 08:33:32 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 17 Apr 2019 07:33:32 -0600
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
	<938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
Message-ID: <874l6wlpsj.fsf@jedbrown.org>

"Smith, Barry F." <bsmith at mcs.anl.gov> writes:

>   This is fine for "hacking" on PETSc but worthless for any other package. Here is my concern, when someone 
> realizes there is a problem with a package they are using through a package manager they think, crud I have to
>
> 1) find the git repository for this package
> 2) git clone the package 
> 3) figure out how to build the package from source, is it ./configure, cmake, what are the needed arguments,... 
> 4) wait for the entire thing to build 
>
> then I can go in and investigate the problem and provide and test the fix via a pull request. Heck I'm not going to bother.

Thanks for explaining.  Yes, this could certainly be improved.  Spack
could run a git diff after a successful build and offer it as a patch.
That might help inform us about hacks that people need even if not
intended for acceptance as-is.

From sajidsyed2021 at u.northwestern.edu  Wed Apr 17 09:25:12 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 17 Apr 2019 09:25:12 -0500
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GkwvX8WcoR8cSDaP2MP+N2e1fKbHDa=EHMK-gRLCGgLQQ@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
	<7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
	<CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>
	<CAMYG4GkwvX8WcoR8cSDaP2MP+N2e1fKbHDa=EHMK-gRLCGgLQQ@mail.gmail.com>
Message-ID: <CAOGsD9h1Bg1MoCa6ec3aG-WUFtvYgXjpP6B6qz2WUfATs_akSw@mail.gmail.com>

Hi Matt/Barry,

I've implemented this for 1D-complex-mpi vec and tested it.

Here is the modified source file ->
https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c

Functions definitions at
https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-395

New op at
https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-514

If this looks good, I can extend it to all cases (1/2/3 dims +
real/complex) and add a vecdupliate/vecdestroy pair in the tests.


Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/dd0de41e/attachment.html>

From knepley at gmail.com  Wed Apr 17 09:32:18 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 17 Apr 2019 10:32:18 -0400
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAOGsD9h1Bg1MoCa6ec3aG-WUFtvYgXjpP6B6qz2WUfATs_akSw@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
	<7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
	<CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>
	<CAMYG4GkwvX8WcoR8cSDaP2MP+N2e1fKbHDa=EHMK-gRLCGgLQQ@mail.gmail.com>
	<CAOGsD9h1Bg1MoCa6ec3aG-WUFtvYgXjpP6B6qz2WUfATs_akSw@mail.gmail.com>
Message-ID: <CAMYG4GmBKA1hskv0SL7XkfkoY3Vq8V7_2sXzTyw8194g9aYzzA@mail.gmail.com>

On Wed, Apr 17, 2019 at 10:25 AM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Hi Matt/Barry,
>
> I've implemented this for 1D-complex-mpi vec and tested it.
>
> Here is the modified source file ->
> https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c
>
> Functions definitions at
> https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-395
>
> New op at
> https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-514
>
> If this looks good, I can extend it to all cases (1/2/3 dims +
> real/complex) and add a vecdupliate/vecdestroy pair in the tests.
>

This is very good. Some suggestions:

1) Send the branch pointer. Its easier for us to look at.

2) Name the branch <username>/fix_<description>

3) Don't like the source files modified in the comment. The ChangeSet tells
us that.

and maybe

4) Squash out some of the sets that are just fixing bugs. However, this is
advanced, so not this time.

  Thanks,

     Matt


> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/5db3435e/attachment.html>

From marian.greg.007 at gmail.com  Wed Apr 17 10:27:15 2019
From: marian.greg.007 at gmail.com (Marian Greg)
Date: Wed, 17 Apr 2019 10:27:15 -0500
Subject: [petsc-users] Iterative solver behavior with increasing number of
 mpi
Message-ID: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>

Hi All,

I am facing strange behavior of the ksp solvers with increasing number of
MPI. The solver is taking more and more iterations with increase in number
of MPIs. Is that a normal situation? I was expecting to get the same number
of iteration with whatever number of MPIs I use.

E.g.
My matrix has about 2 million dofs
Solving with np 1 takes about 3500 iteration while solving with np 4 takes
6500 iterations for the same convergence criteria.

Thanks
Mari
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/44517130/attachment.html>

From balay at mcs.anl.gov  Wed Apr 17 10:39:05 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 17 Apr 2019 15:39:05 +0000
Subject: [petsc-users] Iterative solver behavior with increasing number
 of mpi
In-Reply-To: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>
References: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1904171035120.2713@sb>

Yes - the default preconditioner is block-jacobi - with one block on
each processor.

So when run on 1 proc vs 8 proc - the preconditioner is different
(with 1block for bjacobi vs 8blocks for bjacobi)- hence difference in
convergence.

Satish

On Wed, 17 Apr 2019, Marian Greg via petsc-users wrote:

> Hi All,
> 
> I am facing strange behavior of the ksp solvers with increasing number of
> MPI. The solver is taking more and more iterations with increase in number
> of MPIs. Is that a normal situation? I was expecting to get the same number
> of iteration with whatever number of MPIs I use.
> 
> E.g.
> My matrix has about 2 million dofs
> Solving with np 1 takes about 3500 iteration while solving with np 4 takes
> 6500 iterations for the same convergence criteria.
> 
> Thanks
> Mari
> 


From marian.greg.007 at gmail.com  Wed Apr 17 10:57:10 2019
From: marian.greg.007 at gmail.com (Marian Greg)
Date: Wed, 17 Apr 2019 10:57:10 -0500
Subject: [petsc-users] Iterative solver behavior with increasing number
 of mpi
In-Reply-To: <alpine.LFD.2.21.1904171035120.2713@sb>
References: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>
	<alpine.LFD.2.21.1904171035120.2713@sb>
Message-ID: <CA+uk+nUWSs3MDmiwZ_QV2D_4gbBYeTC2Enww8tSpB_t8DRsi=A@mail.gmail.com>

Thanks Satish for the reply. However, I also observed the same behavior
with gamg and sor preconditioners and ksp_type bcgs as well as gmres. Could
you tell which solver and preconditioners would behave same on whatever
number of mpi I use?

Thanks, Mari


On Wednesday, April 17, 2019, Balay, Satish <balay at mcs.anl.gov> wrote:

> Yes - the default preconditioner is block-jacobi - with one block on
> each processor.
>
> So when run on 1 proc vs 8 proc - the preconditioner is different
> (with 1block for bjacobi vs 8blocks for bjacobi)- hence difference in
> convergence.
>
> Satish
>
> On Wed, 17 Apr 2019, Marian Greg via petsc-users wrote:
>
> > Hi All,
> >
> > I am facing strange behavior of the ksp solvers with increasing number of
> > MPI. The solver is taking more and more iterations with increase in
> number
> > of MPIs. Is that a normal situation? I was expecting to get the same
> number
> > of iteration with whatever number of MPIs I use.
> >
> > E.g.
> > My matrix has about 2 million dofs
> > Solving with np 1 takes about 3500 iteration while solving with np 4
> takes
> > 6500 iterations for the same convergence criteria.
> >
> > Thanks
> > Mari
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/e5253a8e/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Apr 17 11:25:17 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Apr 2019 16:25:17 +0000
Subject: [petsc-users] VecView to hdf5 broken for large (complex) vectors
In-Reply-To: <CAMYG4Gkw0cGineGHtohkE0OHzL4_hvDYjgYo-tA7JppZv_JoWw@mail.gmail.com>
References: <CAOGsD9jfMHjrDBCqSgPL7Qjr43QjyNGMV=C1dJGk5NMT1vAymw@mail.gmail.com>
	<CAMYG4GnLoC-ovgN5w__9TjMjmEiafhPiHPi=RB4HZaabFa8o9g@mail.gmail.com>
	<CAOGsD9jyOCWk4qmRm1owgf7MfMOzQAAg+CjmM3iQoUcsktJiew@mail.gmail.com>
	<CAMYG4Gk8wDvzKVRxE0ij2HoxcBYJ+OK+nZndHC7neM9paJW2bA@mail.gmail.com>
	<CAOGsD9hUF7LJ8BPBy3DHh0FU0X_x3scWc6eqBseH_Z01dWNVyg@mail.gmail.com>
	<8EBCBC08-2151-4334-941B-A52478F26C7C@anl.gov>
	<4EE14CD2-850E-4FC1-AF92-D8536056C85C@anl.gov>
	<CAOGsD9j6=n=5r-m=kCP3EQk_iAwaCdT4ROmJ1atsZk0tR8LvoA@mail.gmail.com>
	<757704F2-508D-42C9-8D38-547B5D86CEF5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162212000.2713@sb>
	<CAOGsD9hZ4ms3Y3kh-QETY5Et2xxv6z8u=YN-ZyKUw_rEpYayUw@mail.gmail.com>
	<76968055-FFA8-4736-AC1C-0F9D8B0808B5@mcs.anl.gov>
	<alpine.LFD.2.21.1904162259040.2713@sb>
	<alpine.LFD.2.21.1904162305550.2713@sb>
	<CAOGsD9j_v9aRJr-Zcf_cgKfigJgykMmt3qne9bur6LH45DYzSA@mail.gmail.com>
	<alpine.LFD.2.21.1904162348580.2713@sb>
	<3F09D019-B847-4A5D-9BEF-26F253DB0246@anl.gov>
	<87a7gpkwe3.fsf@jedbrown.org>
	<938A92E5-D2D4-4CF2-9FB3-718BA8F3E4E7@mcs.anl.gov>
	<alpine.LFD.2.21.1904170129360.2713@sb>
	<BCE816D1-F76A-416D-BAEB-0926A2A01FC8@mcs.anl.gov>
	<CAMYG4Gkw0cGineGHtohkE0OHzL4_hvDYjgYo-tA7JppZv_JoWw@mail.gmail.com>
Message-ID: <FDDCFB36-6E9D-4604-A923-B33651D2968F@mcs.anl.gov>



> On Apr 17, 2019, at 6:49 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Wed, Apr 17, 2019 at 2:40 AM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> > On Apr 17, 2019, at 1:35 AM, Balay, Satish <balay at mcs.anl.gov> wrote:
> > 
> > On Wed, 17 Apr 2019, Smith, Barry F. via petsc-users wrote:
> > 
> >>  This is fine for "hacking" on PETSc but worthless for any other package. Here is my concern, when someone 
> >> realizes there is a problem with a package they are using through a package manager they think, crud I have to
> >> 
> >> 1) find the git repository for this package
> >> 2) git clone the package 
> >> 3) figure out how to build the package from source, is it ./configure, cmake, what are the needed arguments,... 
> >> 4) wait for the entire thing to build 
> >> 
> >> then I can go in and investigate the problem and provide and test the fix via a pull request. Heck I'm not going to bother.
> >> 
> >> Thus a lot of potential contributions of small fixes that everyone in the community would benefit from are lost. This is why, for 
> >> me, an ideal HPC package manager provides a trivial process for providing fixes/improvements to other packages.
> >> 
> >> For example Sajid could have easily figured out the VecView_MPI_HDF5() bug and provided a fix but just the hassle of 
> >> logistics (not ability to solve the problem) prevented him from providing the bug fix to everyone rapidly. 
> > 
> 
>    I never said that any current practices are better than using spack! It is just that perhaps 
> with a few tweaks spack could provide a way to fundamentally improve our current practices (which are, as you acknowledge cumbersome).
> 
> This sounds like a hopeless pipedream. But maybe.

   Most of the infrastructure is there already. 

> 
>    Matt
>  
>    Barry
> 
> > Even without spack and multiple packages - this is not a easy thing to
> > do. For ex: most of our users install petsc from tarball.
> > 
> > And if they find a bug - they have to go through similar complicated
> > process [create a bitbucket account, get a fork - learn the petsc PR
> > process - make a PR etc].
> > 
> > With spack - I stick to the usual process - and don't get bogged down
> > by 'spack' support for this process.
> > 
> > If I see a breakage - I do 'spack build-env package [this has its own
> > issues] - attempt a fix - get it first working with a spack build.
> > 
> > [Alternative is to just edit the package file to get my fix - if its a patch I can find]
> > 
> > 
> > Once I have it working [the major issue is taken care off]. Then I
> > have a diff/patch and then worry about how to submit this diff/patch
> > to upstream.
> > 
> > Sure its a multi step model - and has many trip points. But is not
> > that our current petsc only model doesn't have any.
> > 
> > Satish
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From knepley at gmail.com  Wed Apr 17 11:35:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 17 Apr 2019 12:35:16 -0400
Subject: [petsc-users] Iterative solver behavior with increasing number
 of mpi
In-Reply-To: <CA+uk+nUWSs3MDmiwZ_QV2D_4gbBYeTC2Enww8tSpB_t8DRsi=A@mail.gmail.com>
References: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>
	<alpine.LFD.2.21.1904171035120.2713@sb>
	<CA+uk+nUWSs3MDmiwZ_QV2D_4gbBYeTC2Enww8tSpB_t8DRsi=A@mail.gmail.com>
Message-ID: <CAMYG4GnHEFv11WPMHXykhfQ_Yz21qygrmNkPNzc=7WCFJhkAYw@mail.gmail.com>

On Wed, Apr 17, 2019 at 11:59 AM Marian Greg via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thanks Satish for the reply. However, I also observed the same behavior
> with gamg and sor preconditioners and ksp_type bcgs as well as gmres. Could
> you tell which solver and preconditioners would behave same on whatever
> number of mpi I use?


1) SOR is parallel will also be Block Jacobi-SOR

2) Jacobi will be invariant

3) Chebyshev will be invariant

4) GAMG will be invariant if you have an elliptic equation. So for instance
you can use GAMG on SNES ex5 or ex12 and the iterates will not increase

  Thanks,

    Matt


> Thanks, Mari
>
>
> On Wednesday, April 17, 2019, Balay, Satish <balay at mcs.anl.gov> wrote:
>
>> Yes - the default preconditioner is block-jacobi - with one block on
>> each processor.
>>
>> So when run on 1 proc vs 8 proc - the preconditioner is different
>> (with 1block for bjacobi vs 8blocks for bjacobi)- hence difference in
>> convergence.
>>
>> Satish
>>
>> On Wed, 17 Apr 2019, Marian Greg via petsc-users wrote:
>>
>> > Hi All,
>> >
>> > I am facing strange behavior of the ksp solvers with increasing number
>> of
>> > MPI. The solver is taking more and more iterations with increase in
>> number
>> > of MPIs. Is that a normal situation? I was expecting to get the same
>> number
>> > of iteration with whatever number of MPIs I use.
>> >
>> > E.g.
>> > My matrix has about 2 million dofs
>> > Solving with np 1 takes about 3500 iteration while solving with np 4
>> takes
>> > 6500 iterations for the same convergence criteria.
>> >
>> > Thanks
>> > Mari
>> >
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/2bb16beb/attachment.html>

From mfadams at lbl.gov  Wed Apr 17 15:19:44 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 17 Apr 2019 16:19:44 -0400
Subject: [petsc-users] Iterative solver behavior with increasing number
 of mpi
In-Reply-To: <CAMYG4GnHEFv11WPMHXykhfQ_Yz21qygrmNkPNzc=7WCFJhkAYw@mail.gmail.com>
References: <CA+uk+nWKUAYK+rF4Gg4EvkXoo3cQfERmM+Tjitbn3TAPY_B6vQ@mail.gmail.com>
	<alpine.LFD.2.21.1904171035120.2713@sb>
	<CA+uk+nUWSs3MDmiwZ_QV2D_4gbBYeTC2Enww8tSpB_t8DRsi=A@mail.gmail.com>
	<CAMYG4GnHEFv11WPMHXykhfQ_Yz21qygrmNkPNzc=7WCFJhkAYw@mail.gmail.com>
Message-ID: <CADOhEh7u122qBdsdOKU+nmPGcUDa1ELLZw_nqtRKkTg+0e46vA@mail.gmail.com>

GAMG is almost algorithmically invariant but the graph coarsening is not
invariant not deterministic. You should not see much difference in teration
could but a little decay is expected.

On Wed, Apr 17, 2019 at 12:36 PM Matthew Knepley via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> On Wed, Apr 17, 2019 at 11:59 AM Marian Greg via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Thanks Satish for the reply. However, I also observed the same behavior
>> with gamg and sor preconditioners and ksp_type bcgs as well as gmres. Could
>> you tell which solver and preconditioners would behave same on whatever
>> number of mpi I use?
>
>
> 1) SOR is parallel will also be Block Jacobi-SOR
>
> 2) Jacobi will be invariant
>
> 3) Chebyshev will be invariant
>
> 4) GAMG will be invariant if you have an elliptic equation. So for
> instance you can use GAMG on SNES ex5 or ex12 and the iterates will not
> increase
>
>   Thanks,
>
>     Matt
>
>
>> Thanks, Mari
>>
>>
>> On Wednesday, April 17, 2019, Balay, Satish <balay at mcs.anl.gov> wrote:
>>
>>> Yes - the default preconditioner is block-jacobi - with one block on
>>> each processor.
>>>
>>> So when run on 1 proc vs 8 proc - the preconditioner is different
>>> (with 1block for bjacobi vs 8blocks for bjacobi)- hence difference in
>>> convergence.
>>>
>>> Satish
>>>
>>> On Wed, 17 Apr 2019, Marian Greg via petsc-users wrote:
>>>
>>> > Hi All,
>>> >
>>> > I am facing strange behavior of the ksp solvers with increasing number
>>> of
>>> > MPI. The solver is taking more and more iterations with increase in
>>> number
>>> > of MPIs. Is that a normal situation? I was expecting to get the same
>>> number
>>> > of iteration with whatever number of MPIs I use.
>>> >
>>> > E.g.
>>> > My matrix has about 2 million dofs
>>> > Solving with np 1 takes about 3500 iteration while solving with np 4
>>> takes
>>> > 6500 iterations for the same convergence criteria.
>>> >
>>> > Thanks
>>> > Mari
>>> >
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190417/f26158cf/attachment.html>

From yyang85 at stanford.edu  Thu Apr 18 10:06:25 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 18 Apr 2019 15:06:25 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
	<DM6PR02MB524218828CAD251F331DBC52D2240@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <DM6PR02MB5242A6551F4D8010F0F0BACFD2260@DM6PR02MB5242.namprd02.prod.outlook.com>

Just wanted to follow up on this question, thanks for your time!

Best,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
_____________________________
From: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Sent: Tuesday, April 16, 2019 21:56
Subject: RE: [petsc-users] Using -malloc_dump to examine memory leak
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>, Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>


So using -objects_dump showed nothing below the line:
The following objects were never freed
-----------------------------------------
Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?

I also ran valgrind, and am getting the following error when using --track-origins=yes:
Uninitialised value was created by a stack allocation
==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)

The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?

Thanks for your help on this.

Best regards,
Yuyun

From: petsc-users <petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>> On Behalf Of Yuyun Yang via petsc-users
Sent: Tuesday, April 16, 2019 7:30 AM
To: Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak

Great, thank you for the advice!

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>>
Sent: Tuesday, April 16, 2019 5:54:15 AM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak


  Please try the flag -options_dump this tries to give a much more concise view of what objects have not been freed. For example I commented
out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:

./ex19 -objects_dump
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
The following objects were never freed
-----------------------------------------
[0] DM da DM_0x84000000_0
      [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
[0] Vec seq Vec_0x84000000_6
      [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
      [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c

Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free.

Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
For example because the unfreed vector holds a reference to the DM the DM is listed as not freed. Once you properly destroy the vector you'll
not that the DM is no longer listed as non freed.

It can be a little overwhelming at first to figure out what objects have not been freed. We recommending setting the environmental variable
export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from
the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.

   Good luck



   Barry


> On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hello team,
>
> I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
>
> [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
>       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
>       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
>       [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c
>
> The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
>
> How can I use this option to debug effectively?
>
> Thanks a lot,
> Yuyun


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190418/9060869e/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 18 10:36:09 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 18 Apr 2019 15:36:09 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <DM6PR02MB5242A6551F4D8010F0F0BACFD2260@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
	<DM6PR02MB524218828CAD251F331DBC52D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB5242A6551F4D8010F0F0BACFD2260@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <8E207764-B164-4402-A4F0-DD35881A19EA@mcs.anl.gov>


   Please see my mail from yesterday; perhaps it did not get through

 On Apr 16, 2019, at 11:56 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> 
> So using -objects_dump showed nothing below the line:
> The following objects were never freed
> -----------------------------------------
> Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?

 No idea. If you have messages from -malloc_dump then memory has not been freed and almost always this is due to undestroyed objects. Presumably you didn't turn off logging in ./configure (nobody does that). If you use the option -objects_dump this is processed in PetscOptionsCheckInitial_Private() which is called from PetscInitialize() with 
#if defined(PETSC_USE_LOG)
 ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&PetscObjectsLog);CHKERRQ(ierr);
#endif
then in inherit.c each object creation gets logged with

PetscErrorCode  PetscHeaderCreate_Private(PetscObject h,PetscClassId classid,const char class_name[],const char descr[],const char mansec[],
                                         MPI_Comm comm,PetscObjectDestroyFunction destroy,PetscObjectViewFunction view)
{
 static PetscInt idcnt = 1;
 PetscErrorCode  ierr;
#if defined(PETSC_USE_LOG)
 PetscObject     *newPetscObjects;
 PetscInt         newPetscObjectsMaxCounts,i;
#endif

 PetscFunctionBegin;
 h->classid               = classid;
 h->type                  = 0;
 h->class_name            = (char*)class_name;
 h->description           = (char*)descr;
 h->mansec                = (char*)mansec;
 h->prefix                = 0;
 h->refct                 = 1;
#if defined(PETSC_HAVE_SAWS)
 h->amsmem                = PETSC_FALSE;
#endif
 h->id                    = idcnt++;
 h->parentid              = 0;
 h->qlist                 = 0;
 h->olist                 = 0;
 h->bops->destroy         = destroy;
 h->bops->view            = view;
 h->bops->getcomm         = PetscObjectGetComm_Petsc;
 h->bops->compose         = PetscObjectCompose_Petsc;
 h->bops->query           = PetscObjectQuery_Petsc;
 h->bops->composefunction = PetscObjectComposeFunction_Petsc;
 h->bops->queryfunction   = PetscObjectQueryFunction_Petsc;

 ierr = PetscCommDuplicate(comm,&h->comm,&h->tag);CHKERRQ(ierr);

#if defined(PETSC_USE_LOG)
 /* Keep a record of object created */
 if (PetscObjectsLog) {
   PetscObjectsCounts++;
   for (i=0; i<PetscObjectsMaxCounts; i++) {
     if (!PetscObjects[i]) {
       PetscObjects[i] = h;
       PetscFunctionReturn(0);
     }
   }
   /* Need to increase the space for storing PETSc objects */
   if (!PetscObjectsMaxCounts) newPetscObjectsMaxCounts = 100;
   else                        newPetscObjectsMaxCounts = 2*PetscObjectsMaxCounts;
   ierr = PetscMalloc1(newPetscObjectsMaxCounts,&newPetscObjects);CHKERRQ(ierr);
   ierr = PetscMemcpy(newPetscObjects,PetscObjects,PetscObjectsMaxCounts*sizeof(PetscObject));CHKERRQ(ierr);
   ierr = PetscMemzero(newPetscObjects+PetscObjectsMaxCounts,(newPetscObjectsMaxCounts - PetscObjectsMaxCounts)*sizeof(PetscObject));CHKERRQ(ierr);
   ierr = PetscFree(PetscObjects);CHKERRQ(ierr);

   PetscObjects                        = newPetscObjects;
   PetscObjects[PetscObjectsMaxCounts] = h;
   PetscObjectsMaxCounts               = newPetscObjectsMaxCounts;
 }
#endif
 PetscFunctionReturn(0);
}

Then in PetscFinalize() 

#if defined(PETSC_USE_LOG)
 /*
      List all objects the user may have forgot to free
 */
 if (PetscObjectsLog) {
   ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&flg1);CHKERRQ(ierr);
   if (flg1) {
     MPI_Comm local_comm;
     char     string[64];

     ierr = PetscOptionsGetString(NULL,NULL,"-objects_dump",string,64,NULL);CHKERRQ(ierr);
     ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr);
     ierr = PetscSequentialPhaseBegin_Private(local_comm,1);CHKERRQ(ierr);
     ierr = PetscObjectsDump(stdout,(string[0] == 'a') ? PETSC_TRUE : PETSC_FALSE);CHKERRQ(ierr);
     ierr = PetscSequentialPhaseEnd_Private(local_comm,1);CHKERRQ(ierr);
     ierr = MPI_Comm_free(&local_comm);CHKERRQ(ierr);
   }
 }
#endif

Very simple code. 

Try 

-objects_dump all 

and see if that changes things. Otherwise you'll need to use the debugger to track why it is not printing out the undestroyed objects.


> 
> I also ran valgrind, and am getting the following error when using --track-origins=yes:
> Uninitialised value was created by a stack allocation
> ==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)
> 
> The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?

 Should be unrelated to the -malloc_dump message.

 A "stack allocation" is usually a local variable, while the value from VecGetArray() is from the heap. Check the use of all your local variables in this function.

 Barry


> On Apr 18, 2019, at 10:06 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> Just wanted to follow up on this question, thanks for your time!
> 
> Best,
> Yuyun
> 
> Get Outlook for iOS
> _____________________________
> From: Yuyun Yang <yyang85 at stanford.edu>
> Sent: Tuesday, April 16, 2019 21:56
> Subject: RE: [petsc-users] Using -malloc_dump to examine memory leak
> To: Yuyun Yang <yyang85 at stanford.edu>, Smith, Barry F. <bsmith at mcs.anl.gov>
> 
> 
> So using -objects_dump showed nothing below the line:
> The following objects were never freed
> -----------------------------------------
> Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?
>  
> I also ran valgrind, and am getting the following error when using --track-origins=yes:
> Uninitialised value was created by a stack allocation
> ==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)
>  
> The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?
>  
> Thanks for your help on this.
>  
> Best regards,
> Yuyun
>  
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Yuyun Yang via petsc-users
> Sent: Tuesday, April 16, 2019 7:30 AM
> To: Smith, Barry F. <bsmith at mcs.anl.gov>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> Great, thank you for the advice!
>  
> Best regards,
> Yuyun
>  
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, April 16, 2019 5:54:15 AM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> 
>   Please try the flag -options_dump this tries to give a much more concise view of what objects have not been freed. For example I commented
> out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:
> 
> ./ex19 -objects_dump
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> The following objects were never freed
> -----------------------------------------
> [0] DM da DM_0x84000000_0
>       [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
>       [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> [0] Vec seq Vec_0x84000000_6
>       [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
>       [0]  main() in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> 
> Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free.
> 
> Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
> For example because the unfreed vector holds a reference to the DM the DM is listed as not freed. Once you properly destroy the vector you'll
> not that the DM is no longer listed as non freed.
> 
> It can be a little overwhelming at first to figure out what objects have not been freed. We recommending setting the environmental variable
> export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from
> the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.
> 
>    Good luck
> 
> 
> 
>    Barry
> 
> 
> > On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello team,
> >  
> > I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
> >  
> > [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
> >       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
> >       [0]  VecDuplicate() line 375 in /home/yyy910805/petsc/src/vec/vec/interface/vector.c
> >  
> > The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
> >  
> > How can I use this option to debug effectively?
> >  
> > Thanks a lot,
> > Yuyun
> 


From yyang85 at stanford.edu  Thu Apr 18 10:37:58 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 18 Apr 2019 15:37:58 +0000
Subject: [petsc-users] Using -malloc_dump to examine memory leak
In-Reply-To: <8E207764-B164-4402-A4F0-DD35881A19EA@mcs.anl.gov>
References: <DM6PR02MB5242F95FD66A815269AA6696D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<693DFCAF-05BD-40CE-8390-C57BB39EA45D@anl.gov>
	<DM6PR02MB524218828CAD251F331DBC52D2240@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB524202F2924C8DDD689215C9D2250@DM6PR02MB5242.namprd02.prod.outlook.com>
	<DM6PR02MB5242A6551F4D8010F0F0BACFD2260@DM6PR02MB5242.namprd02.prod.outlook.com>
	<8E207764-B164-4402-A4F0-DD35881A19EA@mcs.anl.gov>
Message-ID: <DM6PR02MB5242CF1A2A71504B5F3D4DF6D2260@DM6PR02MB5242.namprd02.prod.outlook.com>

Oh thank you, sorry I probably missed that!

-----Original Message-----
From: Smith, Barry F. <bsmith at mcs.anl.gov> 
Sent: Thursday, April 18, 2019 8:36 AM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak


   Please see my mail from yesterday; perhaps it did not get through

 On Apr 16, 2019, at 11:56 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> 
> So using -objects_dump showed nothing below the line:
> The following objects were never freed
> -----------------------------------------
> Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?

 No idea. If you have messages from -malloc_dump then memory has not been freed and almost always this is due to undestroyed objects. Presumably you didn't turn off logging in ./configure (nobody does that). If you use the option -objects_dump this is processed in PetscOptionsCheckInitial_Private() which is called from PetscInitialize() with #if defined(PETSC_USE_LOG)  ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&PetscObjectsLog);CHKERRQ(ierr);
#endif
then in inherit.c each object creation gets logged with

PetscErrorCode  PetscHeaderCreate_Private(PetscObject h,PetscClassId classid,const char class_name[],const char descr[],const char mansec[],
                                         MPI_Comm comm,PetscObjectDestroyFunction destroy,PetscObjectViewFunction view) {  static PetscInt idcnt = 1;  PetscErrorCode  ierr; #if defined(PETSC_USE_LOG)
 PetscObject     *newPetscObjects;
 PetscInt         newPetscObjectsMaxCounts,i;
#endif

 PetscFunctionBegin;
 h->classid               = classid;
 h->type                  = 0;
 h->class_name            = (char*)class_name;
 h->description           = (char*)descr;
 h->mansec                = (char*)mansec;
 h->prefix                = 0;
 h->refct                 = 1;
#if defined(PETSC_HAVE_SAWS)
 h->amsmem                = PETSC_FALSE;
#endif
 h->id                    = idcnt++;
 h->parentid              = 0;
 h->qlist                 = 0;
 h->olist                 = 0;
 h->bops->destroy         = destroy;
 h->bops->view            = view;
 h->bops->getcomm         = PetscObjectGetComm_Petsc;
 h->bops->compose         = PetscObjectCompose_Petsc;
 h->bops->query           = PetscObjectQuery_Petsc;
 h->bops->composefunction = PetscObjectComposeFunction_Petsc;
 h->bops->queryfunction   = PetscObjectQueryFunction_Petsc;

 ierr = PetscCommDuplicate(comm,&h->comm,&h->tag);CHKERRQ(ierr);

#if defined(PETSC_USE_LOG)
 /* Keep a record of object created */
 if (PetscObjectsLog) {
   PetscObjectsCounts++;
   for (i=0; i<PetscObjectsMaxCounts; i++) {
     if (!PetscObjects[i]) {
       PetscObjects[i] = h;
       PetscFunctionReturn(0);
     }
   }
   /* Need to increase the space for storing PETSc objects */
   if (!PetscObjectsMaxCounts) newPetscObjectsMaxCounts = 100;
   else                        newPetscObjectsMaxCounts = 2*PetscObjectsMaxCounts;
   ierr = PetscMalloc1(newPetscObjectsMaxCounts,&newPetscObjects);CHKERRQ(ierr);
   ierr = PetscMemcpy(newPetscObjects,PetscObjects,PetscObjectsMaxCounts*sizeof(PetscObject));CHKERRQ(ierr);
   ierr = PetscMemzero(newPetscObjects+PetscObjectsMaxCounts,(newPetscObjectsMaxCounts - PetscObjectsMaxCounts)*sizeof(PetscObject));CHKERRQ(ierr);
   ierr = PetscFree(PetscObjects);CHKERRQ(ierr);

   PetscObjects                        = newPetscObjects;
   PetscObjects[PetscObjectsMaxCounts] = h;
   PetscObjectsMaxCounts               = newPetscObjectsMaxCounts;
 }
#endif
 PetscFunctionReturn(0);
}

Then in PetscFinalize() 

#if defined(PETSC_USE_LOG)
 /*
      List all objects the user may have forgot to free  */  if (PetscObjectsLog) {
   ierr = PetscOptionsHasName(NULL,NULL,"-objects_dump",&flg1);CHKERRQ(ierr);
   if (flg1) {
     MPI_Comm local_comm;
     char     string[64];

     ierr = PetscOptionsGetString(NULL,NULL,"-objects_dump",string,64,NULL);CHKERRQ(ierr);
     ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr);
     ierr = PetscSequentialPhaseBegin_Private(local_comm,1);CHKERRQ(ierr);
     ierr = PetscObjectsDump(stdout,(string[0] == 'a') ? PETSC_TRUE : PETSC_FALSE);CHKERRQ(ierr);
     ierr = PetscSequentialPhaseEnd_Private(local_comm,1);CHKERRQ(ierr);
     ierr = MPI_Comm_free(&local_comm);CHKERRQ(ierr);
   }
 }
#endif

Very simple code. 

Try 

-objects_dump all 

and see if that changes things. Otherwise you'll need to use the debugger to track why it is not printing out the undestroyed objects.


> 
> I also ran valgrind, and am getting the following error when using --track-origins=yes:
> Uninitialised value was created by a stack allocation
> ==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)
> 
> The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?

 Should be unrelated to the -malloc_dump message.

 A "stack allocation" is usually a local variable, while the value from VecGetArray() is from the heap. Check the use of all your local variables in this function.

 Barry


> On Apr 18, 2019, at 10:06 AM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> Just wanted to follow up on this question, thanks for your time!
> 
> Best,
> Yuyun
> 
> Get Outlook for iOS
> _____________________________
> From: Yuyun Yang <yyang85 at stanford.edu>
> Sent: Tuesday, April 16, 2019 21:56
> Subject: RE: [petsc-users] Using -malloc_dump to examine memory leak
> To: Yuyun Yang <yyang85 at stanford.edu>, Smith, Barry F. 
> <bsmith at mcs.anl.gov>
> 
> 
> So using -objects_dump showed nothing below the line:
> The following objects were never freed
> -----------------------------------------
> Does that mean I?ve freed all the objects? If so, why would -malloc_dump give me a bunch of messages?
>  
> I also ran valgrind, and am getting the following error when using --track-origins=yes:
> Uninitialised value was created by a stack allocation
> ==617==    at 0x417D66: ComputeVel_qd::computeVel(double*, double, int&, int) (fault.cpp:953)
>  
> The first argument of this function is a Petsc array (obtained from VecGetArray, and after calling this function, VecRestoreArray was called on that same object). I can?t think of a way why valgrind thinks there is an uninitialized value. Is this by any chance related to the -malloc_dump messages?
>  
> Thanks for your help on this.
>  
> Best regards,
> Yuyun
>  
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Yuyun 
> Yang via petsc-users
> Sent: Tuesday, April 16, 2019 7:30 AM
> To: Smith, Barry F. <bsmith at mcs.anl.gov>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> Great, thank you for the advice!
>  
> Best regards,
> Yuyun
>  
> Get Outlook for iOS
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, April 16, 2019 5:54:15 AM
> To: Yuyun Yang
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Using -malloc_dump to examine memory leak
>  
> 
>   Please try the flag -options_dump this tries to give a much more 
> concise view of what objects have not been freed. For example I commented out the last VecDestroy() in src/snes/examples/tutorials/ex19.c and then obtained:
> 
> ./ex19 -objects_dump
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> The following objects were never freed
> -----------------------------------------
> [0] DM da DM_0x84000000_0
>       [0]  DMDACreate2d() in /Users/barrysmith/Src/petsc/src/dm/impls/da/da2.c
>       [0]  main() in 
> /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> [0] Vec seq Vec_0x84000000_6
>       [0]  DMCreateGlobalVector() in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c
>       [0]  main() in 
> /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c
> 
> Now I just need to look at the calls to DMCreateGlobalVector and DMDACreate2d in main to see what I did not free.
> 
> Note that since PETSc objects may hold references to other PETSc objects some items may not be freed for which you DID call destroy on.
> For example because the unfreed vector holds a reference to the DM the 
> DM is listed as not freed. Once you properly destroy the vector you'll not that the DM is no longer listed as non freed.
> 
> It can be a little overwhelming at first to figure out what objects 
> have not been freed. We recommending setting the environmental 
> variable export PETSC_OPTIONS=-malloc_test so that every run of your code reports memory issues and you can keep them under control from the beginning (when the code is small and growing) rather than wait until the code is large and there are many unfreed objects to chase down.
> 
>    Good luck
> 
> 
> 
>    Barry
> 
> 
> > On Apr 16, 2019, at 1:14 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hello team,
> >  
> > I?m trying to use the options -malloc_dump and -malloc_debug to examine memory leaks. The messages however, are quite generic, and don?t really tell me where the problems occur, for example:
> >  
> > [ 0]1520 bytes VecCreate() line 35 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  PetscMallocA() line 35 in /home/yyy910805/petsc/src/sys/memory/mal.c
> >       [0]  VecCreate() line 30 in /home/yyy910805/petsc/src/vec/vec/interface/veccreate.c
> >       [0]  VecDuplicate_Seq() line 804 in /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c
> >       [0]  VecDuplicate() line 375 in 
> > /home/yyy910805/petsc/src/vec/vec/interface/vector.c
> >  
> > The code is huge, so going through every single VecCreate/VecDuplicate and VecDestroy is going to be time-consuming. Meanwhile, running valgrind gave me some uninitialized values errors that don?t seem to be related to the above message (or maybe they are?).
> >  
> > How can I use this option to debug effectively?
> >  
> > Thanks a lot,
> > Yuyun
> 


From yyang85 at stanford.edu  Thu Apr 18 10:44:08 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 18 Apr 2019 15:44:08 +0000
Subject: [petsc-users] Question on KSPSetReusePreconditioner
Message-ID: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I have a question on when I should be setting KSPSetReusePreconditioner to PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions I get when I ran my code testing both settings, so is recalculating the preconditioner better since it will make the convergence faster and the solution more accurate? On the other hand, if the operator is updated every time step (but same nonzero pattern), will reconstructing the pc become time-consuming?

Thank you!
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190418/24c4cdd6/attachment.html>

From knepley at gmail.com  Thu Apr 18 11:32:18 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 18 Apr 2019 12:32:18 -0400
Subject: [petsc-users] Question on KSPSetReusePreconditioner
In-Reply-To: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4Gk7LRUNUZdNC3hA83NOGoJNyNzbi+PtXnx-iBxdM3XjFA@mail.gmail.com>

On Thu, Apr 18, 2019 at 11:44 AM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello team,
>
>
>
> I have a question on when I should be setting KSPSetReusePreconditioner to
> PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions
> I get when I ran my code testing both settings, so is recalculating the
> preconditioner better since it will make the convergence faster and the
> solution more accurate? On the other hand, if the operator is updated every
> time step (but same nonzero pattern), will reconstructing the pc become
> time-consuming?
>

We cannot say anything since it (at least) depends on a) how much your
matrix changes from one solve to the next, and b)  what preconditioner you
are reusing.

  Thanks,

    Matt


>
>
> Thank you!
>
> Yuyun
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190418/156ad72c/attachment-0001.html>

From yyang85 at stanford.edu  Thu Apr 18 13:46:39 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 18 Apr 2019 18:46:39 +0000
Subject: [petsc-users] Question on KSPSetReusePreconditioner
In-Reply-To: <CAMYG4Gk7LRUNUZdNC3hA83NOGoJNyNzbi+PtXnx-iBxdM3XjFA@mail.gmail.com>
References: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4Gk7LRUNUZdNC3hA83NOGoJNyNzbi+PtXnx-iBxdM3XjFA@mail.gmail.com>
Message-ID: <DM6PR02MB5242B7140D1876B93217485FD2260@DM6PR02MB5242.namprd02.prod.outlook.com>

Let?s say I?m using HYPRE?s boomeramg preconditioner. Could we say anything about convergence/pc construction time if the matrix changes significantly from one solve to the next vs. slightly changes?

Thanks,
Yuyun

From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, April 18, 2019 9:32 AM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question on KSPSetReusePreconditioner

On Thu, Apr 18, 2019 at 11:44 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I have a question on when I should be setting KSPSetReusePreconditioner to PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions I get when I ran my code testing both settings, so is recalculating the preconditioner better since it will make the convergence faster and the solution more accurate? On the other hand, if the operator is updated every time step (but same nonzero pattern), will reconstructing the pc become time-consuming?

We cannot say anything since it (at least) depends on a) how much your matrix changes from one solve to the next, and b)  what preconditioner you are reusing.

  Thanks,

    Matt


Thank you!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190418/80449ecd/attachment.html>

From knepley at gmail.com  Thu Apr 18 14:10:46 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 18 Apr 2019 15:10:46 -0400
Subject: [petsc-users] Question on KSPSetReusePreconditioner
In-Reply-To: <DM6PR02MB5242B7140D1876B93217485FD2260@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4Gk7LRUNUZdNC3hA83NOGoJNyNzbi+PtXnx-iBxdM3XjFA@mail.gmail.com>
	<DM6PR02MB5242B7140D1876B93217485FD2260@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4G=wq8eUuJ5JogpVTvNt-owCZjj196cu1t3PAJ7zs4c5Jg@mail.gmail.com>

On Thu, Apr 18, 2019 at 2:46 PM Yuyun Yang <yyang85 at stanford.edu> wrote:

> Let?s say I?m using HYPRE?s boomeramg preconditioner. Could we say
> anything about convergence/pc construction time if the matrix changes
> significantly from one solve to the next vs. slightly changes?
>

Hypre does not store anything (I believe), so it does not matter at all.

  Thanks,

     Matt


>
>
> Thanks,
>
> Yuyun
>
>
>
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, April 18, 2019 9:32 AM
> *To:* Yuyun Yang <yyang85 at stanford.edu>
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Question on KSPSetReusePreconditioner
>
>
>
> On Thu, Apr 18, 2019 at 11:44 AM Yuyun Yang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hello team,
>
>
>
> I have a question on when I should be setting KSPSetReusePreconditioner to
> PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions
> I get when I ran my code testing both settings, so is recalculating the
> preconditioner better since it will make the convergence faster and the
> solution more accurate? On the other hand, if the operator is updated every
> time step (but same nonzero pattern), will reconstructing the pc become
> time-consuming?
>
>
>
> We cannot say anything since it (at least) depends on a) how much your
> matrix changes from one solve to the next, and b)  what preconditioner you
> are reusing.
>
>
>
>   Thanks,
>
>
>
>     Matt
>
>
>
>
>
> Thank you!
>
> Yuyun
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190418/1ae5f5b3/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 18 17:51:27 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 18 Apr 2019 22:51:27 +0000
Subject: [petsc-users] Question on KSPSetReusePreconditioner
In-Reply-To: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <86E6B512-B5A3-4695-9040-9CA4C17B0664@anl.gov>



> On Apr 18, 2019, at 10:44 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello team,
>  
> I have a question on when I should be setting KSPSetReusePreconditioner to PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions I get when I ran my code testing both settings, so is recalculating the preconditioner better since it will make the convergence faster and the solution more accurate?

   Reusing the preconditioner should have no significant impact on the accuracy of the solution to the linear system. The answers will be different but that should only be up to the tolerance you provided with -ksp_rtol   But after many time-steps the final "answer" to the time dependent problem may be very different with different preconditioning approaches due to accumulation of different round-offs. But if both preconditioner approaches solve the linear system to the same accuracy then neither answer is more correct than the other.

> On the other hand, if the operator is updated every time step (but same nonzero pattern), will reconstructing the pc become time-consuming?

  Whether to reuse the preconditioner depends completely on the problem. Reuse means more linear iterations but less setup time (for the preconditioner). So the relative cost of iterations to setup time dictates the optimal use of KSPSetReusePreconditioner() and can only be determined by making some timing runs with different choices and picking the one that seems fastest for your problem.

   Barry

>  
> Thank you!
> Yuyun


From yyang85 at stanford.edu  Thu Apr 18 20:53:36 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Fri, 19 Apr 2019 01:53:36 +0000
Subject: [petsc-users] Question on KSPSetReusePreconditioner
In-Reply-To: <86E6B512-B5A3-4695-9040-9CA4C17B0664@anl.gov>
References: <DM6PR02MB52428E023E6E771B710CE5A0D2260@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<86E6B512-B5A3-4695-9040-9CA4C17B0664@anl.gov>
Message-ID: <BYAPR02MB5240F3CF1D95F280E13F7E37D2270@BYAPR02MB5240.namprd02.prod.outlook.com>

I see, thanks for the helpful answers!

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Thursday, April 18, 2019 3:51:27 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Question on KSPSetReusePreconditioner



> On Apr 18, 2019, at 10:44 AM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Hello team,
>
> I have a question on when I should be setting KSPSetReusePreconditioner to PETSC_TRUE or PETSC_FALSE. It made a pretty big difference in the solutions I get when I ran my code testing both settings, so is recalculating the preconditioner better since it will make the convergence faster and the solution more accurate?

   Reusing the preconditioner should have no significant impact on the accuracy of the solution to the linear system. The answers will be different but that should only be up to the tolerance you provided with -ksp_rtol   But after many time-steps the final "answer" to the time dependent problem may be very different with different preconditioning approaches due to accumulation of different round-offs. But if both preconditioner approaches solve the linear system to the same accuracy then neither answer is more correct than the other.

> On the other hand, if the operator is updated every time step (but same nonzero pattern), will reconstructing the pc become time-consuming?

  Whether to reuse the preconditioner depends completely on the problem. Reuse means more linear iterations but less setup time (for the preconditioner). So the relative cost of iterations to setup time dictates the optimal use of KSPSetReusePreconditioner() and can only be determined by making some timing runs with different choices and picking the one that seems fastest for your problem.

   Barry

>
> Thank you!
> Yuyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190419/e2d9f67e/attachment-0001.html>

From zonexo at gmail.com  Sun Apr 21 08:27:22 2019
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 21 Apr 2019 21:27:22 +0800
Subject: [petsc-users] Does PetscLogEventBegin and PetscLogEventEnd slow
 down the code if it's called every time step?
Message-ID: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>

Hi,

I'm trying to do some logging as well as timing of my code?

May I know if PetscLogEventBegin and PetscLogEventEnd slow down my code 
if it's called every time step?

-- 

Thank you very much.

Yours sincerely,

================================================
TAY Wee-Beng ??? (Zheng Weiming)
Personal research webpage: _http://tayweebeng.wixsite.com/website_
Youtube research showcase: _https://goo.gl/PtvdwQ_
linkedin: _https://www.linkedin.com/in/tay-weebeng_
================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190421/054681a0/attachment.html>

From knepley at gmail.com  Sun Apr 21 08:43:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 21 Apr 2019 09:43:04 -0400
Subject: [petsc-users] Does PetscLogEventBegin and PetscLogEventEnd slow
 down the code if it's called every time step?
In-Reply-To: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>
References: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>
Message-ID: <CAMYG4Gk2PQ=hckBrb-_Q9HQ9E_be8Ezv=DReuQW5A3gAFJ2DHA@mail.gmail.com>

On Sun, Apr 21, 2019 at 9:28 AM TAY wee-beng via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I'm trying to do some logging as well as timing of my code?
>
> May I know if PetscLogEventBegin and PetscLogEventEnd slow down my code if
> it's called every time step?
>
At the granularity of a timestep should not be a problem. At the
granularity of every entry in a Jacobian, it could
be a problem.

  Thanks,

     MAtt

> --
>
> Thank you very much.
>
> Yours sincerely,
>
> ================================================
> TAY Wee-Beng ??? (Zheng Weiming)
> Personal research webpage: *http://tayweebeng.wixsite.com/website
> <http://tayweebeng.wixsite.com/website>*
> Youtube research showcase: *https://goo.gl/PtvdwQ <https://goo.gl/PtvdwQ>*
> linkedin: *https://www.linkedin.com/in/tay-weebeng
> <https://www.linkedin.com/in/tay-weebeng>*
> ================================================
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190421/19244cfb/attachment.html>

From koshmipt at gmail.com  Sun Apr 21 17:28:22 2019
From: koshmipt at gmail.com (Oleksandr Koshkarov)
Date: Sun, 21 Apr 2019 16:28:22 -0600
Subject: [petsc-users] DMDASetBlockFillsSparse format
Message-ID: <CACuBWh1j5-zHqi1vyRzf8Vc7o61orqw3t3ZWxezpRwOT6Un5zQ@mail.gmail.com>

Dear All,
Does anyone know if the format of sparse matrices
in DMDASetBlockFillsSparse require column indicies for chosen row to be in
increasing order?
Thank you,
Oleksandr.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190421/608b79c2/attachment.html>

From jed at jedbrown.org  Sun Apr 21 17:42:26 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 21 Apr 2019 16:42:26 -0600
Subject: [petsc-users] DMDASetBlockFillsSparse format
In-Reply-To: <CACuBWh1j5-zHqi1vyRzf8Vc7o61orqw3t3ZWxezpRwOT6Un5zQ@mail.gmail.com>
References: <CACuBWh1j5-zHqi1vyRzf8Vc7o61orqw3t3ZWxezpRwOT6Un5zQ@mail.gmail.com>
Message-ID: <87k1fndlpp.fsf@jedbrown.org>

I'm pretty confident all the tests are sorted.  It wouldn't be any great
hardship for us to allow unsorted input.  If you submit an unsorted
test, we can make sure it works (it might already, but we should
probably add a call to in-place sort).

Oleksandr Koshkarov via petsc-users <petsc-users at mcs.anl.gov> writes:

> Dear All,
> Does anyone know if the format of sparse matrices
> in DMDASetBlockFillsSparse require column indicies for chosen row to be in
> increasing order?
> Thank you,
> Oleksandr.

From koshkarov.alexandr at usask.ca  Sun Apr 21 17:46:33 2019
From: koshkarov.alexandr at usask.ca (Koshkarov, Oleksandr)
Date: Sun, 21 Apr 2019 22:46:33 +0000
Subject: [petsc-users] DMDASetBlockFillsSparse format
Message-ID: <YQXPR01MB25035BB3ADD92E8A9659F261F0210@YQXPR01MB2503.CANPRD01.PROD.OUTLOOK.COM>

Dear All,
Does anyone know if the format of sparse matrices in DMDASetBlockFillsSparse require column indicies for chosen row to be in increasing order?
Thank you,
Oleksandr.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190421/b464f6ba/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr 22 00:33:39 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Apr 2019 05:33:39 +0000
Subject: [petsc-users] DMDASetBlockFillsSparse format
In-Reply-To: <87k1fndlpp.fsf@jedbrown.org>
References: <CACuBWh1j5-zHqi1vyRzf8Vc7o61orqw3t3ZWxezpRwOT6Un5zQ@mail.gmail.com>
	<87k1fndlpp.fsf@jedbrown.org>
Message-ID: <51EA86C3-48A2-46F9-AD35-CBBEF9EECC4E@anl.gov>


  I looked at the code and in the end the values get propagated into calls to MatSetValuesLocal() which does not require sorted values. 
  Thus my conclusion is that you do not have to sorted column indices in these calls

   Barry

   Yes we do a nonexistent job of indicating when indices need to be sorted. Most places: creation of IS, MatSetValuesXXX, VecSetValuesXXX do not need to be sorted. We should mark explicitly any locations where int array values should be sorted in the API, if it is not marked you should assume they do not need to be sorted.

> On Apr 21, 2019, at 5:42 PM, Jed Brown via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> I'm pretty confident all the tests are sorted.  It wouldn't be any great
> hardship for us to allow unsorted input.  If you submit an unsorted
> test, we can make sure it works (it might already, but we should
> probably add a call to in-place sort).
> 
> Oleksandr Koshkarov via petsc-users <petsc-users at mcs.anl.gov> writes:
> 
>> Dear All,
>> Does anyone know if the format of sparse matrices
>> in DMDASetBlockFillsSparse require column indicies for chosen row to be in
>> increasing order?
>> Thank you,
>> Oleksandr.


From bsmith at mcs.anl.gov  Mon Apr 22 00:56:07 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Apr 2019 05:56:07 +0000
Subject: [petsc-users] Does PetscLogEventBegin and PetscLogEventEnd slow
 down the code if it's called every time step?
In-Reply-To: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>
References: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>
Message-ID: <450A5993-960D-4155-B9BC-953095BDC25D@anl.gov>



> On Apr 21, 2019, at 8:27 AM, TAY wee-beng via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi,
> 
> I'm trying to do some logging as well as timing of my code?
> 
> May I know if PetscLogEventBegin and PetscLogEventEnd slow down my code if it's called every time step?

   The logging is only done if the -log_view option is set or PetscLogDefaultBegin() is called. 

   Make two identical runs on the same machine once with -log_view, once without and time them. (use unix time on a work station and perhaps the patch system you use provides runtime information). Make the machines have the same load (or lack of load) on them to get a fair comparison. It would be very surprising if the -log_view version had a nontrivial effect on the run times.

  Barry

> 
> -- 
> Thank you very much.
> 
> Yours sincerely,
> 
> ================================================
> TAY Wee-Beng ??? (Zheng Weiming)
> Personal research webpage: http://tayweebeng.wixsite.com/website
> Youtube research showcase: https://goo.gl/PtvdwQ
> linkedin: https://www.linkedin.com/in/tay-weebeng
> ================================================
> 


From bsmith at mcs.anl.gov  Mon Apr 22 01:16:23 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Apr 2019 06:16:23 +0000
Subject: [petsc-users] Error with VecDestroy_MPIFFTW+0x61
In-Reply-To: <CAMYG4GmBKA1hskv0SL7XkfkoY3Vq8V7_2sXzTyw8194g9aYzzA@mail.gmail.com>
References: <CAOGsD9gAFr8Raoo47t4G=2O5ihbrWpZNOAtTvcuyVUZXiDFw8A@mail.gmail.com>
	<CAMYG4G=J1DWGKR6evW506E-FBas1rPLNRQhG6_1+YY8NQAerEQ@mail.gmail.com>
	<CAOGsD9ihXjYUksDmV=TeE22i-rSmTFGip=jkhZW3hoCxU_8Jwg@mail.gmail.com>
	<CAMYG4GmOA8ewz-q=npE3AF3ADYqKAoeZB_sSXOMzp508MSrVWQ@mail.gmail.com>
	<CAOGsD9gXtznL=vfrhLP1C8Ri1ke9_y775U5eKNT4qRkjsaZ6CQ@mail.gmail.com>
	<CAMYG4GknewkROiS4HAKb9VM2m2NVo6BmPsdJrKs50tVaaQM+Uw@mail.gmail.com>
	<1B93C6F9-3E9E-48E0-82C7-34D22FB8E342@anl.gov>
	<CAMYG4GmbN5ztJVy78TZgZ6aAPfSj-YQtAc3ikZdsQnmJDEfo+A@mail.gmail.com>
	<CAOGsD9jfNCVq6hLXL1Cb90O_zWY3QOsFt9ZpgHru+2=osaSB9w@mail.gmail.com>
	<C7C15090-3C27-46A8-B992-6E96A9C7F943@mcs.anl.gov>
	<CAMYG4GmPZmo88LCiVUhh3DvjFycmhOa7p0B7F=-QbMB-0_mZ5g@mail.gmail.com>
	<7F8B53AA-1C27-4A5E-87AE-76C75FDAA7BD@mcs.anl.gov>
	<CAOGsD9hF2SNCnGufM+aHbNg7CoppD_0+tPO73ZUUeFe3a21zXg@mail.gmail.com>
	<CAMYG4GkwvX8WcoR8cSDaP2MP+N2e1fKbHDa=EHMK-gRLCGgLQQ@mail.gmail.com>
	<CAOGsD9h1Bg1MoCa6ec3aG-WUFtvYgXjpP6B6qz2WUfATs_akSw@mail.gmail.com>
	<CAMYG4GmBKA1hskv0SL7XkfkoY3Vq8V7_2sXzTyw8194g9aYzzA@mail.gmail.com>
Message-ID: <F0207B33-8C7F-4F74-A3F2-FF686EB409F6@mcs.anl.gov>


  Sajid,

    Where is the pull request for this? Better to get in the Bibuckit pull request system then have it drag out and have code change around. So please go ahead and post the pull request. It looked pretty much done the last time I saw it.

    Barry


> On Apr 17, 2019, at 9:32 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Wed, Apr 17, 2019 at 10:25 AM Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> Hi Matt/Barry, 
> 
> I've implemented this for 1D-complex-mpi vec and tested it. 
> 
> Here is the modified source file -> https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c
> 
> Functions definitions at https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-395
> 
> New op at https://bitbucket.org/sajid__ali/petsc/src/86fb19b57a7c4f8f42644e5160d2afbdc5e03639/src/mat/impls/fft/fftw/fftw.c#lines-514
> 
> If this looks good, I can extend it to all cases (1/2/3 dims + real/complex) and add a vecdupliate/vecdestroy pair in the tests. 
> 
> This is very good. Some suggestions:
> 
> 1) Send the branch pointer. Its easier for us to look at.
> 
> 2) Name the branch <username>/fix_<description>
> 
> 3) Don't like the source files modified in the comment. The ChangeSet tells us that.
> 
> and maybe
> 
> 4) Squash out some of the sets that are just fixing bugs. However, this is advanced, so not this time.
> 
>   Thanks,
> 
>      Matt
>  
> Thank You, 
> Sajid Ali
> Applied Physics
> Northwestern University
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From eda.oktay at metu.edu.tr  Mon Apr 22 03:28:03 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Mon, 22 Apr 2019 11:28:03 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
Message-ID: <CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>

Thank you for your answers. I figured out that for a matrix of size
4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
wrote the program such that the local sizes of IS 2127 and 2126.

Is local size of Mat being 2127 on both processors correct? If it is, then
I will change the local size of IS but then it will exceed the global size
of Mat. Isn't this also a problem?

Thanks a lot,

Eda

Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde ?unu
yazd?:

>
>   Suggest printing out the IS with ISView after it is created and
> confirming that 1) it is a permutation and 2) that the size of the IS on
> each process matches the number of rows on that process.
>
>   Note from the manual page: The index sets should be on the same
> communicator as Mat and have the same local sizes.
>
> Barry
>
>
> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello again,
> >
> > I solved the problem for even numbered sized matrices. However when the
> matrix size is odd, then number of elements in each index set at each
> processor are different. (For example, for size 4253*4253 and 2 processors,
> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
> I think this is why, MatPermute again gives the same "Argument out of
> range" error. Index sets look like correct but I still did not get why I
> get this error.
> >
> > This is the part of my program:
> >
> >   PetscMPIInt rank,size;
> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
> >
> >   PetscInt mod;
> >   mod = siz % size;
> >
> >   PetscInt *idxx,ss;
> >   ss = (siz-mod)/size;
>
> >
> >   if (mod != 0){
> >     if (rank<mod){
> >         PetscMalloc1(ss+1,&idxx);
> >     } else{
> >         PetscMalloc1(ss,&idxx);
> >     }
> >   }
> >
> >   if (rank != size-1) {
> >     j =0;
> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
> >       idxx[j] = idx[i];
> >       j++;
> >     }
> >
> >   } else {
> >
> >     j =0;
> >     for (i=rank*ss; i<siz; i++) {
> >       idxx[j] = idx[i];
> >       j++;
> >     }
> >
> >   }
> >
> >   if (mod != 0){
> >     if (rank<mod){
> >         idxx[ss+1] = idx[ss*size+rank+1];
> >     }
> >   }
> >
> >   /*Permute matrix L (spy(A(p1,p1))*/
> >
> >     if (mod != 0){
> >     if (rank<mod){
> >         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
> >     } else{
> >         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
> >     }
> >   }
> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
> >
> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
> >
> > And I get the following error even if I use MatSetOption :
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Argument out of range
> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
> 11:10:59 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #5 main() line 352 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> > [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > [0]PETSC ERROR: -mat_partitioning_type parmetis
> > [0]PETSC ERROR: -unweighted
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> >
> > Thanks!
> >
> > Eda
> >
> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
> ?unu yazd?:
> > I attached whole program I wrote where the problem is in line 285. One
> of the matrices I used was airfoil1_binary, included in the folder. Also, I
> included makefile. Is that what you want?
> >
> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde
> ?unu yazd?:
> > That should not happen. Can you send in a small example that we can
> debug.
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hello,
> >
> > I am trying to permute a vector A using following lines:
> >
> > ierr =
> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
> >   ierr =
> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
> >
> > However, in MatPermute line, I get the following error even if I used
> MatSetOption before this line:
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Argument out of range
> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #6 main() line 285 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
> > [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > [0]PETSC ERROR: -mat_partitioning_type parmetis
> > [0]PETSC ERROR: -weighted
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> >
> > I'll be glad if you can help me.
> >
> > Thanks!
> >
> > Eda
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/70b5222f/attachment-0001.html>

From eda.oktay at metu.edu.tr  Mon Apr 22 07:31:00 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Mon, 22 Apr 2019 15:31:00 +0300
Subject: [petsc-users] Memory Corruption Error in MatPartitioningApply
Message-ID: <CA+vu_XgNSDDXA3JE6KjpAwRM3df+-m7ms-V7trTB29=afyE-CA@mail.gmail.com>

Hello,

I am trying to partition an odd-numbered sized (for example 4253*4253),
square permutation matrix by using 2 processors with ParMETIS. The
permutation matrix is obtained by permuting the matrix by an index set "is"
(MatPermute(A,is,is,&PL)). I checked the index set, it gives a permutation
and it is correct.

When I look at the local size of the matrix, it is given by 2127 and 2127
on each processor, so in order the local sizes of matrix and index sets to
be same, I defined the index sets' sizes as 2127 and 2127.

When I do that, I get memory corruption error in MatPartiitioningApply
function. The error is as follows:

[0]PETSC ERROR: PetscMallocValidate: error detected at
MatPartitioningApply_Parmetis_Private() line 141 in
/home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
[0]PETSC ERROR: Memory [id=0(8512)] at address 0x19e6870 is corrupted
(probably write past end of array)
[0]PETSC ERROR: Memory originally allocated in main() line 310 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Memory corruption:
http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind
[0]PETSC ERROR:
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
arch-linux2-c-debug named 13ed.wls.metu.edu.tr by edaoktay Mon Apr 22
14:58:52 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 PetscMallocValidate() line 146 in
/home/edaoktay/petsc-3.10.3/src/sys/memory/mtr.c
[0]PETSC ERROR: #2 MatPartitioningApply_Parmetis_Private() line 141 in
/home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
[0]PETSC ERROR: #3 MatPartitioningApply_Parmetis() line 215 in
/home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
[0]PETSC ERROR: #4 MatPartitioningApply() line 340 in
/home/edaoktay/petsc-3.10.3/src/mat/partition/partition.c
[0]PETSC ERROR: #5 main() line 374 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -f
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
[0]PETSC ERROR: -mat_partitioning_type parmetis
[0]PETSC ERROR: -unweighted
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------


The line 310 is PetscMalloc1(ss,&idxx). The part of my program is written
as below:

  if (mod != 0){
      ss = (siz+1)/size;//(siz+size-mod)/size;
  } else{
      ss = siz/size;
  }

PetscMalloc1(ss,&idxx);                                       // LINE 310

  if (rank != size-1) {
    j =0;
    for (i=rank*ss; i<(rank+1)*ss; i++) {
      idxx[j] = idx[i];
      j++;
    }

  } else {

    j =0;
    for (i=rank*ss; i<siz; i++) {
      idxx[j] = idx[i];
      j++;
    }

  }

  if (mod != 0){
    if (rank<mod){
        idxx[ss+1] = idx[ss*size+rank+1];
    }
  }

  /*Permute matrix L (spy(A(p1,p1))*/

    if (mod != 0){
        if (rank<mod){
            ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
        } else{
            ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
        }

    }else {
        ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
    }

  ierr = ISSetPermutation(is);CHKERRQ(ierr);
  ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);


    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                    Create Partitioning
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  ierr = MatConvert(PL,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr);

  ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr);
  ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr);

  ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
  ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr);

I understood that I cannot change the local size of the matrix since it is
read from a file. But as you can see above, when I defined index sets'
sizes as 2127 and 2127, memory corruption occurs. I tried several things
but at the end I got error in MatPermute or here.

By the way, idx is from 0 to 4252 but the global size of is is 4253. If I
change idx to 0:4253 then I think it will be incorrect since actually there
is no 4254th element.

How can I solve this problem?

Thank you,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/b402a718/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr 22 08:47:19 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Apr 2019 13:47:19 +0000
Subject: [petsc-users] Memory Corruption Error in MatPartitioningApply
In-Reply-To: <CA+vu_XgNSDDXA3JE6KjpAwRM3df+-m7ms-V7trTB29=afyE-CA@mail.gmail.com>
References: <CA+vu_XgNSDDXA3JE6KjpAwRM3df+-m7ms-V7trTB29=afyE-CA@mail.gmail.com>
Message-ID: <B7B3E6D9-4D85-4E32-87B5-314DE8445813@anl.gov>


  Are you able to run under valgrind? It is a bit better than the PETSc malloc to find each instance of memory corruption and the sooner you find it the easier it is to find the bug. https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind



> On Apr 22, 2019, at 7:31 AM, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
> 
> I am trying to partition an odd-numbered sized (for example 4253*4253), square permutation matrix by using 2 processors with ParMETIS. The permutation matrix is obtained by permuting the matrix by an index set "is" (MatPermute(A,is,is,&PL)). I checked the index set, it gives a permutation and it is correct.
> 
> When I look at the local size of the matrix, it is given by 2127 and 2127 on each processor, so in order the local sizes of matrix and index sets to be same, I defined the index sets' sizes as 2127 and 2127. 
> 
> When I do that, I get memory corruption error in MatPartiitioningApply function. The error is as follows:
> 
> [0]PETSC ERROR: PetscMallocValidate: error detected at MatPartitioningApply_Parmetis_Private() line 141 in /home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
> [0]PETSC ERROR: Memory [id=0(8512)] at address 0x19e6870 is corrupted (probably write past end of array)
> [0]PETSC ERROR: Memory originally allocated in main() line 310 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Memory corruption: http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind
> [0]PETSC ERROR:  
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named 13ed.wls.metu.edu.tr by edaoktay Mon Apr 22 14:58:52 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [0]PETSC ERROR: #1 PetscMallocValidate() line 146 in /home/edaoktay/petsc-3.10.3/src/sys/memory/mtr.c
> [0]PETSC ERROR: #2 MatPartitioningApply_Parmetis_Private() line 141 in /home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
> [0]PETSC ERROR: #3 MatPartitioningApply_Parmetis() line 215 in /home/edaoktay/petsc-3.10.3/src/mat/partition/impls/pmetis/pmetis.c
> [0]PETSC ERROR: #4 MatPartitioningApply() line 340 in /home/edaoktay/petsc-3.10.3/src/mat/partition/partition.c
> [0]PETSC ERROR: #5 main() line 374 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 
> 
> The line 310 is PetscMalloc1(ss,&idxx). The part of my program is written as below:
> 
>   if (mod != 0){
>       ss = (siz+1)/size;//(siz+size-mod)/size;
>   } else{
>       ss = siz/size;
>   }
>   
> PetscMalloc1(ss,&idxx);                                       // LINE 310
> 
>   if (rank != size-1) {
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
> 
>   } else {
>       
>     j =0;
>     for (i=rank*ss; i<siz; i++) {
>       idxx[j] = idx[i];
>       j++;
>     }
>       
>   }
>    
>   if (mod != 0){
>     if (rank<mod){
>         idxx[ss+1] = idx[ss*size+rank+1];
>     }
>   }
> 
>   /*Permute matrix L (spy(A(p1,p1))*/
>   
>     if (mod != 0){
>         if (rank<mod){
>             ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         } else{
>             ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         } 
>         
>     }else {
>         ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     }
> 
>   ierr = ISSetPermutation(is);CHKERRQ(ierr); 
>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);                                
> 
>     /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>                     Create Partitioning
>      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
>   
>   ierr = MatConvert(PL,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr);                    
>   ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr);
>   ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr);                             
>   ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
>   ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr);        
> 
> I understood that I cannot change the local size of the matrix since it is read from a file. But as you can see above, when I defined index sets' sizes as 2127 and 2127, memory corruption occurs. I tried several things but at the end I got error in MatPermute or here.
> 
> By the way, idx is from 0 to 4252 but the global size of is is 4253. If I change idx to 0:4253 then I think it will be incorrect since actually there is no 4254th element.
> 
> How can I solve this problem?
> 
> Thank you,
> 
> Eda


From stefano.zampini at gmail.com  Mon Apr 22 10:13:37 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 22 Apr 2019 18:13:37 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
Message-ID: <CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>

If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
default local sizes should be 2127 and 2126.

Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

> Thank you for your answers. I figured out that for a matrix of size
> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
> wrote the program such that the local sizes of IS 2127 and 2126.
>
> Is local size of Mat being 2127 on both processors correct? If it is, then
> I will change the local size of IS but then it will exceed the global size
> of Mat. Isn't this also a problem?
>
> Thanks a lot,
>
> Eda
>
> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
> ?unu yazd?:
>
>>
>>   Suggest printing out the IS with ISView after it is created and
>> confirming that 1) it is a permutation and 2) that the size of the IS on
>> each process matches the number of rows on that process.
>>
>>   Note from the manual page: The index sets should be on the same
>> communicator as Mat and have the same local sizes.
>>
>> Barry
>>
>>
>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Hello again,
>> >
>> > I solved the problem for even numbered sized matrices. However when the
>> matrix size is odd, then number of elements in each index set at each
>> processor are different. (For example, for size 4253*4253 and 2 processors,
>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>> I think this is why, MatPermute again gives the same "Argument out of
>> range" error. Index sets look like correct but I still did not get why I
>> get this error.
>> >
>> > This is the part of my program:
>> >
>> >   PetscMPIInt rank,size;
>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>> >
>> >   PetscInt mod;
>> >   mod = siz % size;
>> >
>> >   PetscInt *idxx,ss;
>> >   ss = (siz-mod)/size;
>>
>> >
>> >   if (mod != 0){
>> >     if (rank<mod){
>> >         PetscMalloc1(ss+1,&idxx);
>>
>> >     } else{
>> >         PetscMalloc1(ss,&idxx);
>> >     }
>> >   }
>> >
>> >   if (rank != size-1) {
>> >     j =0;
>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>> >       idxx[j] = idx[i];
>> >       j++;
>> >     }
>> >
>> >   } else {
>> >
>> >     j =0;
>> >     for (i=rank*ss; i<siz; i++) {
>> >       idxx[j] = idx[i];
>> >       j++;
>> >     }
>> >
>> >   }
>> >
>> >   if (mod != 0){
>> >     if (rank<mod){
>> >         idxx[ss+1] = idx[ss*size+rank+1];
>> >     }
>> >   }
>> >
>> >   /*Permute matrix L (spy(A(p1,p1))*/
>> >
>> >     if (mod != 0){
>> >     if (rank<mod){
>> >         ierr =
>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>> >     } else{
>> >         ierr =
>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>> >     }
>> >   }
>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>> >
>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>> >
>> > And I get the following error even if I use MatSetOption :
>> >
>> > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [0]PETSC ERROR: Argument out of range
>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>> turn off this check
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
>> 11:10:59 2019
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [0]PETSC ERROR: #5 main() line 352 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [0]PETSC ERROR: PETSc Option Table entries:
>> > [0]PETSC ERROR: -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>> > [0]PETSC ERROR: -unweighted
>> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> >
>> > Thanks!
>> >
>> > Eda
>> >
>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
>> ?unu yazd?:
>> > I attached whole program I wrote where the problem is in line 285. One
>> of the matrices I used was airfoil1_binary, included in the folder. Also, I
>> included makefile. Is that what you want?
>> >
>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde
>> ?unu yazd?:
>> > That should not happen. Can you send in a small example that we can
>> debug.
>> >
>> >   Thanks,
>> >
>> >     Matt
>> >
>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > Hello,
>> >
>> > I am trying to permute a vector A using following lines:
>> >
>> > ierr =
>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>> >   ierr =
>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>> >
>> > However, in MatPermute line, I get the following error even if I used
>> MatSetOption before this line:
>> >
>> > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [0]PETSC ERROR: Argument out of range
>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>> turn off this check
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [0]PETSC ERROR: #6 main() line 285 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>> > [0]PETSC ERROR: PETSc Option Table entries:
>> > [0]PETSC ERROR: -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>> > [0]PETSC ERROR: -weighted
>> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> >
>> > I'll be glad if you can help me.
>> >
>> > Thanks!
>> >
>> > Eda
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/40f9bff0/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Mon Apr 22 17:00:38 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 22 Apr 2019 17:00:38 -0500
Subject: [petsc-users] Possible Bug ?
Message-ID: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>

Hi,

I see that src/mat/examples/tests/ex112.c is failing for petsc at 3.11.1
configured without complex scalars. With complex scalars everything works
fine.

The error I see is :
```
[sajid at xrmlite bugfix]$
./ex112

No protocol
specified



 1-D: FFTW on vector of size
10

  Error norm of |x - z|
5.37156

  Error norm of |x - z|
5.90871

  Error norm of |x - z|
5.96243

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------

[0]PETSC ERROR: Nonconforming object
sizes

[0]PETSC ERROR: Mat mat,Vec x: global dim 20
10

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12,
2019

[0]PETSC ERROR: ./ex112 on a  named xrmlite.phys.northwestern.edu by sajid
Mon Apr 22 16:58:41 2019
[0]PETSC ERROR: Configure options
--prefix=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/petsc-3.11.1-5bdbcozu3labtbbi7gtq4xa
knay24lo6 --with-ssl=0 --download-c2html=0 --download-sowing=0
--download-hwloc=0 CFLAGS="-O2 -march=native" FFLAGS="-O2 -march=native"
CXXFLAGS="-
O2 -march=native"
--with-cc=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic
c
--with-cxx=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic++
--with-fc=/h
ome/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpif90
--with-precision=double --w
ith-scalar-type=real --with-shared-libraries=1 --with-debugging=1
--with-64-bit-indices=0
--with-blaslapack-lib="/home/sajid/packages/spack/opt/spa
ck/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/li
bmkl_intel_lp64.so
/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compil
ers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_sequential.so
/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-m
kl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_core.so
/lib64/libpthread.so /lib64/
libm.so /lib64/libdl.so" --with-x=0 --with-clanguage=C --with-scalapack=0
--with-metis=0 --with-hdf5=0 --with-hypre=0 --with-parmetis=0 --with-mump
s=0 --with-trilinos=0 --with-fftw=1
--with-fftw-dir=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/fftw-3.3.8-mkj4ho2jp6xfrnkl
mrvhdfh73woer2s7 --with-superlu_dist=0 --with-suitesparse=0
--with-zlib-include=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0
/zlib-1.2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/include
--with-zlib-lib="-L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/zlib-1
.2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/lib -lz"
--with-zlib=1

[0]PETSC ERROR: #1 MatMult() line 2385 in
/tmp/sajid/spack-stage/spack-stage-mTD0AV/petsc-3.11.1/src/mat/interface/matrix.c

[0]PETSC ERROR: #2 main() line 94 in
/home/sajid/packages/aux_xwp_petsc/bugfix/ex112.c

[0]PETSC ERROR: No PETSc Option Table
entries

[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 60) - process
0

[unset]: write_line error; fd=-1 buf=:cmd=abort
exitcode=60

:

system msg for write_line failure : Bad file
descriptor


```

I came across this because I saw that MatMult was failing for a new test
related to a PR I was working on. Is this a bug ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/90bac681/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Mon Apr 22 17:06:48 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 22 Apr 2019 17:06:48 -0500
Subject: [petsc-users] Possible Bug ?
In-Reply-To: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
References: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
Message-ID: <CAOGsD9j6v1euvUGuaFN1kKR4KnNgAnf=jqz5rjYtTBXxx-M_NQ@mail.gmail.com>

Apologies for the post. I didn't see that it was for complex vectors only.

On Mon, Apr 22, 2019 at 5:00 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> Hi,
>
> I see that src/mat/examples/tests/ex112.c is failing for petsc at 3.11.1
> configured without complex scalars. With complex scalars everything works
> fine.
>
> The error I see is :
> ```
> [sajid at xrmlite bugfix]$
> ./ex112
>
> No protocol
> specified
>
>
>
>  1-D: FFTW on vector of size
> 10
>
>   Error norm of |x - z|
> 5.37156
>
>   Error norm of |x - z|
> 5.90871
>
>   Error norm of |x - z|
> 5.96243
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Nonconforming object
> sizes
>
> [0]PETSC ERROR: Mat mat,Vec x: global dim 20
> 10
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12,
> 2019
>
> [0]PETSC ERROR: ./ex112 on a  named xrmlite.phys.northwestern.edu by
> sajid Mon Apr 22 16:58:41
> 2019
> [0]PETSC ERROR: Configure options
> --prefix=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/petsc-3.11.1-5bdbcozu3labtbbi7gtq4xa
> knay24lo6 --with-ssl=0 --download-c2html=0 --download-sowing=0
> --download-hwloc=0 CFLAGS="-O2 -march=native" FFLAGS="-O2 -march=native"
> CXXFLAGS="-
> O2 -march=native"
> --with-cc=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic
> c
> --with-cxx=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic++
> --with-fc=/h
> ome/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpif90
> --with-precision=double --w
> ith-scalar-type=real --with-shared-libraries=1 --with-debugging=1
> --with-64-bit-indices=0
> --with-blaslapack-lib="/home/sajid/packages/spack/opt/spa
>
> ck/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/li
> bmkl_intel_lp64.so
> /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compil
> ers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_sequential.so
> /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-m
> kl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_core.so
> /lib64/libpthread.so /lib64/
> libm.so /lib64/libdl.so" --with-x=0 --with-clanguage=C --with-scalapack=0
> --with-metis=0 --with-hdf5=0 --with-hypre=0 --with-parmetis=0 --with-mump
> s=0 --with-trilinos=0 --with-fftw=1
> --with-fftw-dir=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/fftw-3.3.8-mkj4ho2jp6xfrnkl
> mrvhdfh73woer2s7 --with-superlu_dist=0 --with-suitesparse=0
> --with-zlib-include=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0
> /zlib-1.2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/include
> --with-zlib-lib="-L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/zlib-1
> .2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/lib -lz"
> --with-zlib=1
>
> [0]PETSC ERROR: #1 MatMult() line 2385 in
> /tmp/sajid/spack-stage/spack-stage-mTD0AV/petsc-3.11.1/src/mat/interface/matrix.c
>
> [0]PETSC ERROR: #2 main() line 94 in
> /home/sajid/packages/aux_xwp_petsc/bugfix/ex112.c
>
> [0]PETSC ERROR: No PETSc Option Table
> entries
>
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
>
> application called MPI_Abort(MPI_COMM_WORLD, 60) - process
> 0
>
> [unset]: write_line error; fd=-1 buf=:cmd=abort
> exitcode=60
>
> :
>
> system msg for write_line failure : Bad file
> descriptor
>
>
> ```
>
> I came across this because I saw that MatMult was failing for a new test
> related to a PR I was working on. Is this a bug ?
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/ac1691a9/attachment.html>

From bsmith at mcs.anl.gov  Mon Apr 22 17:10:06 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Apr 2019 22:10:06 +0000
Subject: [petsc-users] Possible Bug ?
In-Reply-To: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
References: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
Message-ID: <30B7CEBA-A836-4048-BC2B-3F344049E8B6@anl.gov>


   I'll take a look at it. Compiling now. I note that this example is currently only run in the tests in complex, not sure why it is not tested in real

   Barry


> On Apr 22, 2019, at 5:00 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi, 
> 
> I see that src/mat/examples/tests/ex112.c is failing for petsc at 3.11.1 configured without complex scalars. With complex scalars everything works fine. 
> 
> The error I see is : 
> ```
> [sajid at xrmlite bugfix]$  ./ex112                                                                                                                    
> No protocol specified                                                                                                                              
>                                                                                                                                                    
>  1-D: FFTW on vector of size 10                                                                                                                    
>   Error norm of |x - z| 5.37156                                                                                                                    
>   Error norm of |x - z| 5.90871                                                                                                                    
>   Error norm of |x - z| 5.96243                                                                                                                    
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------                                 
> [0]PETSC ERROR: Nonconforming object sizes                                                                                                         
> [0]PETSC ERROR: Mat mat,Vec x: global dim 20 10                                                                                                    
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.                                                      
> [0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019                                                                                        
> [0]PETSC ERROR: ./ex112 on a  named xrmlite.phys.northwestern.edu by sajid Mon Apr 22 16:58:41 2019                                                
> [0]PETSC ERROR: Configure options --prefix=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/petsc-3.11.1-5bdbcozu3labtbbi7gtq4xa
> knay24lo6 --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS="-O2 -march=native" FFLAGS="-O2 -march=native" CXXFLAGS="-
> O2 -march=native" --with-cc=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic
> c --with-cxx=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpic++ --with-fc=/h
> ome/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/mpich-3.3-ig4cr2xw2x63bqs5rnmhfshln4iv7av5/bin/mpif90 --with-precision=double --w
> ith-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-64-bit-indices=0 --with-blaslapack-lib="/home/sajid/packages/spack/opt/spa
> ck/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/li
> bmkl_intel_lp64.so /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-mkl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compil
> ers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_sequential.so /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/intel-m
> kl-2019.3.199-kzcly5rtcjbkwtnm6tri6kkexnwoat5m/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_core.so /lib64/libpthread.so /lib64/
> libm.so /lib64/libdl.so" --with-x=0 --with-clanguage=C --with-scalapack=0 --with-metis=0 --with-hdf5=0 --with-hypre=0 --with-parmetis=0 --with-mump
> s=0 --with-trilinos=0 --with-fftw=1 --with-fftw-dir=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/fftw-3.3.8-mkj4ho2jp6xfrnkl
> mrvhdfh73woer2s7 --with-superlu_dist=0 --with-suitesparse=0 --with-zlib-include=/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0
> /zlib-1.2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/include --with-zlib-lib="-L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/zlib-1
> .2.11-jqrcjdjnrxvouufhjtxbfvfms23fsqpx/lib -lz" --with-zlib=1                                                                                      
> [0]PETSC ERROR: #1 MatMult() line 2385 in /tmp/sajid/spack-stage/spack-stage-mTD0AV/petsc-3.11.1/src/mat/interface/matrix.c                        
> [0]PETSC ERROR: #2 main() line 94 in /home/sajid/packages/aux_xwp_petsc/bugfix/ex112.c                                                             
> [0]PETSC ERROR: No PETSc Option Table entries                                                                                                      
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------                         
> application called MPI_Abort(MPI_COMM_WORLD, 60) - process 0                                                                                       
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=60                                                                                        
> :                                                                                                                                                  
> system msg for write_line failure : Bad file descriptor                                                                                            
> 
> ```
> 
> I came across this because I saw that MatMult was failing for a new test related to a PR I was working on. Is this a bug ? 
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University


From sajidsyed2021 at u.northwestern.edu  Mon Apr 22 18:05:13 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 22 Apr 2019 18:05:13 -0500
Subject: [petsc-users] Possible Bug ?
In-Reply-To: <30B7CEBA-A836-4048-BC2B-3F344049E8B6@anl.gov>
References: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
	<30B7CEBA-A836-4048-BC2B-3F344049E8B6@anl.gov>
Message-ID: <CAOGsD9htZSU8KLgd_X32Z=BOi6STAPMwnjedsp-Zmm0MPzKWTg@mail.gmail.com>

Hi Barry,

I'm not sure why MatCreateVecsFFW is not used at line 50/51.

The error occurs at line 94 because in the second loop, the example
manually creates the x vector instead of the one created using the A
matrix. For complex numbers this is not an issue but for real numbers the
dimensions don't match. MatCreateVecsFFTW creates a vector of size 20 for
N=10 but VecCreateSeq is creating a vector of size 10. I'm not sure what
the rationale behind this test is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/52bcf921/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Mon Apr 22 18:46:39 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 22 Apr 2019 18:46:39 -0500
Subject: [petsc-users] Possible Bug ?
In-Reply-To: <CAOGsD9htZSU8KLgd_X32Z=BOi6STAPMwnjedsp-Zmm0MPzKWTg@mail.gmail.com>
References: <CAOGsD9jbfSO1-+n3wY0ZZoXu7CcifDUk1ZsXfnn4_EPx_Zdn=g@mail.gmail.com>
	<30B7CEBA-A836-4048-BC2B-3F344049E8B6@anl.gov>
	<CAOGsD9htZSU8KLgd_X32Z=BOi6STAPMwnjedsp-Zmm0MPzKWTg@mail.gmail.com>
Message-ID: <CAOGsD9jVgeHc21A2NHsrC1u51N6TSphThEcv2c5mwk-YyFFehA@mail.gmail.com>

This can be tracked down to n vs N being used. The vector in the second
loop is created using N while MatCreateVecsFFTW uses n (for real numbers).
n!=N and hence the error.

If the lines 50/51 and line 91 are switched to MatCreateVecsFFW instead of
MatGetVecs and VecCreateSeq respectively, the example would work for real
numbers as well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190422/1d32a675/attachment.html>

From zonexo at gmail.com  Mon Apr 22 21:38:31 2019
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 23 Apr 2019 10:38:31 +0800
Subject: [petsc-users] Does PetscLogEventBegin and PetscLogEventEnd slow
 down the code if it's called every time step?
In-Reply-To: <450A5993-960D-4155-B9BC-953095BDC25D@anl.gov>
References: <a0b6cefe-ed97-5ddd-b893-2751c369ae6c@gmail.com>
	<450A5993-960D-4155-B9BC-953095BDC25D@anl.gov>
Message-ID: <c322e09e-7358-5ce2-a63e-2a74e40da7d5@gmail.com>

Hi Barry and Matt

Thank you very much for the explanation.

Yours sincerely,

================================================
TAY Wee-Beng ??? (Zheng Weiming)
Personal research webpage: _http://tayweebeng.wixsite.com/website_
Youtube research showcase: _https://goo.gl/PtvdwQ_
linkedin: _https://www.linkedin.com/in/tay-weebeng_
================================================

On 22/4/2019 1:56 PM, Smith, Barry F. wrote:
>
>> On Apr 21, 2019, at 8:27 AM, TAY wee-beng via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi,
>>
>> I'm trying to do some logging as well as timing of my code?
>>
>> May I know if PetscLogEventBegin and PetscLogEventEnd slow down my code if it's called every time step?
>     The logging is only done if the -log_view option is set or PetscLogDefaultBegin() is called.
>
>     Make two identical runs on the same machine once with -log_view, once without and time them. (use unix time on a work station and perhaps the patch system you use provides runtime information). Make the machines have the same load (or lack of load) on them to get a fair comparison. It would be very surprising if the -log_view version had a nontrivial effect on the run times.
>
>    Barry
>
>> -- 
>> Thank you very much.
>>
>> Yours sincerely,
>>
>> ================================================
>> TAY Wee-Beng ??? (Zheng Weiming)
>> Personal research webpage: http://tayweebeng.wixsite.com/website
>> Youtube research showcase: https://goo.gl/PtvdwQ
>> linkedin: https://www.linkedin.com/in/tay-weebeng
>> ================================================
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190423/1ad1ff2d/attachment.html>

From maahi.buet at gmail.com  Tue Apr 23 20:55:45 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Tue, 23 Apr 2019 21:55:45 -0400
Subject: [petsc-users] Preallocation of sequential matrix
Message-ID: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>

Dear All,


I am trying to preallocate the no of nonzeros in my matrix using the
parameter 'nnz'. Here 'row' is the parameter 'nnz'. The part of the code
that does that is the following-

..........................................................................................................................................................................................
*Do j = 2,xmax-1*

*Do i = 2,ymax-1*

*a = (ymax-2)*(j-2)+i-1-1*

*If(j.eq.2 .and. i .ge. 3 .and. i .le. (ymax-2))then*
*row(a) = 6*

*else if (j.eq.(xmax-1) .and. i.ge.3 .and. i .le. (ymax-2)) then*
*row(a) = 6*

*else if(i.eq.2 .and. j.ge.3 .and. j.le.(xmax-2))then*
*row(a) = 6*

*else if(i.eq.(ymax-1) .and. j.ge.3 .and. j.le.(xmax-2)) then*
*row(a) = 6*

*else if(i.eq.2 .and. j.eq.2) then*
*row(a) = 4*

*else if (i.eq.2 .and. j .eq. (xmax-1)) then*
*row(a)= 4*

*else if (i.eq.(ymax-1) .and. j .eq. 2) then*
*row(a) = 4*

*else if (i .eq. (ymax-1) .and. j .eq. (xmax-1)) then*
*row(a) = 4*

*else*
*row(a) = 9*

*end if*


*end do *

*end do *


*call MatCreateSeqAIJ(PETSC_COMM_SELF,N,N,ze,row,Mp,ierr)*

.................................................................................................................................................................................................

But I get the following error message :

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (61,124) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr
23 21:39:26 2019
[0]PETSC ERROR: Configure options --with-debugging=0
--download-fblaslapack=1 PETSC_ARCH=arch-opt
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 481 in
/home/maahi/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1349 in
/home/maahi/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (124,186) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr
23 21:39:26 2019
[0]PETSC ERROR: Configure options --with-debugging=0
--download-fblaslapack=1 PETSC_ARCH=arch-opt
[0]PETSC ERROR: #3 MatSetValues_SeqAIJ() line 481 in
/home/maahi/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #4 MatSetValues() line 1349 in
/home/maahi/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
.................................................................................................................................................................................

But instead of using 'nnz', if I put a upper bound for 'nz',  the code
works fine.

Any idea what went wrong?

Thanks,
Maahi Talukder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190423/206099ed/attachment.html>

From knepley at gmail.com  Tue Apr 23 21:05:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Apr 2019 22:05:04 -0400
Subject: [petsc-users] Preallocation of sequential matrix
In-Reply-To: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
References: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
Message-ID: <CAMYG4GnP+YqaHC6OFfnbB5LohoxFagATx-eRZJF3Yf+et9V2Gw@mail.gmail.com>

On Tue, Apr 23, 2019 at 9:57 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear All,
>
>
> I am trying to preallocate the no of nonzeros in my matrix using the
> parameter 'nnz'. Here 'row' is the parameter 'nnz'. The part of the code
> that does that is the following-
>
>
> ..........................................................................................................................................................................................
> *Do j = 2,xmax-1*
>
> *Do i = 2,ymax-1*
>
> *a = (ymax-2)*(j-2)+i-1-1*
>
> *If(j.eq.2 .and. i .ge. 3 .and. i .le. (ymax-2))then*
> *row(a) = 6*
>
> *else if (j.eq.(xmax-1) .and. i.ge.3 .and. i .le. (ymax-2)) then*
> *row(a) = 6*
>
> *else if(i.eq.2 .and. j.ge.3 .and. j.le.(xmax-2))then*
> *row(a) = 6*
>
> *else if(i.eq.(ymax-1) .and. j.ge.3 .and. j.le.(xmax-2)) then*
> *row(a) = 6*
>
> *else if(i.eq.2 .and. j.eq.2) then*
> *row(a) = 4*
>
> *else if (i.eq.2 .and. j .eq. (xmax-1)) then*
> *row(a)= 4*
>
> *else if (i.eq.(ymax-1) .and. j .eq. 2) then*
> *row(a) = 4*
>
> *else if (i .eq. (ymax-1) .and. j .eq. (xmax-1)) then*
> *row(a) = 4*
>
> *else*
> *row(a) = 9*
>
> *end if*
>
>
> *end do *
>
> *end do *
>
>
> *call MatCreateSeqAIJ(PETSC_COMM_SELF,N,N,ze,row,Mp,ierr)*
>
>
> .................................................................................................................................................................................................
>
> But I get the following error message :
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (61,124) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue
> Apr 23 21:39:26 2019
> [0]PETSC ERROR: Configure options --with-debugging=0
> --download-fblaslapack=1 PETSC_ARCH=arch-opt
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 481 in
> /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1349 in
> /home/maahi/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (124,186) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
> off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue
> Apr 23 21:39:26 2019
> [0]PETSC ERROR: Configure options --with-debugging=0
> --download-fblaslapack=1 PETSC_ARCH=arch-opt
> [0]PETSC ERROR: #3 MatSetValues_SeqAIJ() line 481 in
> /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #4 MatSetValues() line 1349 in
> /home/maahi/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> .................................................................................................................................................................................
>
> But instead of using 'nnz', if I put a upper bound for 'nz',  the code
> works fine.
>
> Any idea what went wrong?
>

I am guessing you are not converting to the global row index the same as
PETSc.

   Matt


> Thanks,
> Maahi Talukder
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190423/adde6669/attachment.html>

From jczhang at mcs.anl.gov  Tue Apr 23 22:40:40 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Wed, 24 Apr 2019 03:40:40 +0000
Subject: [petsc-users] Preallocation of sequential matrix
In-Reply-To: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
References: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
Message-ID: <CA+MQGp-Px78RU0LQSk6Hq-rV0D-+HnX8a_0fanE=G_WV3OAGPQ@mail.gmail.com>

The error message has
[0]PETSC ERROR: New nonzero at (61,124) caused a malloc
[0]PETSC ERROR: New nonzero at (124,186) caused a malloc
You can check your code to see if you allocated spots for these nonzeros.

--Junchao Zhang


On Tue, Apr 23, 2019 at 8:57 PM Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear All,


I am trying to preallocate the no of nonzeros in my matrix using the parameter 'nnz'. Here 'row' is the parameter 'nnz'. The part of the code that does that is the following-

..........................................................................................................................................................................................
Do j = 2,xmax-1

Do i = 2,ymax-1

a = (ymax-2)*(j-2)+i-1-1

If(j.eq.2 .and. i .ge. 3 .and. i .le. (ymax-2))then
row(a) = 6

else if (j.eq.(xmax-1) .and. i.ge.3 .and. i .le. (ymax-2)) then
row(a) = 6

else if(i.eq.2 .and. j.ge.3 .and. j.le.(xmax-2))then
row(a) = 6

else if(i.eq.(ymax-1) .and. j.ge.3 .and. j.le.(xmax-2)) then
row(a) = 6

else if(i.eq.2 .and. j.eq.2) then
row(a) = 4

else if (i.eq.2 .and. j .eq. (xmax-1)) then
row(a)= 4

else if (i.eq.(ymax-1) .and. j .eq. 2) then
row(a) = 4

else if (i .eq. (ymax-1) .and. j .eq. (xmax-1)) then
row(a) = 4

else
row(a) = 9

end if


end do

end do


call MatCreateSeqAIJ(PETSC_COMM_SELF,N,N,ze,row,Mp,ierr)

.................................................................................................................................................................................................

But I get the following error message :

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (61,124) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr 23 21:39:26 2019
[0]PETSC ERROR: Configure options --with-debugging=0 --download-fblaslapack=1 PETSC_ARCH=arch-opt
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 481 in /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1349 in /home/maahi/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (124,186) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr 23 21:39:26 2019
[0]PETSC ERROR: Configure options --with-debugging=0 --download-fblaslapack=1 PETSC_ARCH=arch-opt
[0]PETSC ERROR: #3 MatSetValues_SeqAIJ() line 481 in /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #4 MatSetValues() line 1349 in /home/maahi/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
.................................................................................................................................................................................

But instead of using 'nnz', if I put a upper bound for 'nz',  the code works fine.

Any idea what went wrong?

Thanks,
Maahi Talukder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/e28831a3/attachment-0001.html>

From maahi.buet at gmail.com  Tue Apr 23 23:24:38 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Wed, 24 Apr 2019 00:24:38 -0400
Subject: [petsc-users] Preallocation of sequential matrix
In-Reply-To: <CA+MQGp-Px78RU0LQSk6Hq-rV0D-+HnX8a_0fanE=G_WV3OAGPQ@mail.gmail.com>
References: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
	<CA+MQGp-Px78RU0LQSk6Hq-rV0D-+HnX8a_0fanE=G_WV3OAGPQ@mail.gmail.com>
Message-ID: <CAJOaupEA6M6OxNTKYyaP=aJjQmWwk7ADtKB5ZBNoKS5ypMyqXg@mail.gmail.com>

Thank you all. I will look into that.

Maahi

On Tue, Apr 23, 2019, 11:40 PM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

> The error message has
>
> [0]PETSC ERROR: New nonzero at (61,124) caused a malloc
> [0]PETSC ERROR: New nonzero at (124,186) caused a malloc
>
> You can check your code to see if you allocated spots for these nonzeros.
>
> --Junchao Zhang
>
>
> On Tue, Apr 23, 2019 at 8:57 PM Maahi Talukder via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear All,
>>
>>
>> I am trying to preallocate the no of nonzeros in my matrix using the
>> parameter 'nnz'. Here 'row' is the parameter 'nnz'. The part of the code
>> that does that is the following-
>>
>>
>> ..........................................................................................................................................................................................
>> *Do j = 2,xmax-1*
>>
>> *Do i = 2,ymax-1*
>>
>> *a = (ymax-2)*(j-2)+i-1-1*
>>
>> *If(j.eq.2 .and. i .ge. 3 .and. i .le. (ymax-2))then*
>> *row(a) = 6*
>>
>> *else if (j.eq.(xmax-1) .and. i.ge.3 .and. i .le. (ymax-2)) then*
>> *row(a) = 6*
>>
>> *else if(i.eq.2 .and. j.ge.3 .and. j.le.(xmax-2))then*
>> *row(a) = 6*
>>
>> *else if(i.eq.(ymax-1) .and. j.ge.3 .and. j.le.(xmax-2)) then*
>> *row(a) = 6*
>>
>> *else if(i.eq.2 .and. j.eq.2) then*
>> *row(a) = 4*
>>
>> *else if (i.eq.2 .and. j .eq. (xmax-1)) then*
>> *row(a)= 4*
>>
>> *else if (i.eq.(ymax-1) .and. j .eq. 2) then*
>> *row(a) = 4*
>>
>> *else if (i .eq. (ymax-1) .and. j .eq. (xmax-1)) then*
>> *row(a) = 4*
>>
>> *else*
>> *row(a) = 9*
>>
>> *end if*
>>
>>
>> *end do *
>>
>> *end do *
>>
>>
>> *call MatCreateSeqAIJ(PETSC_COMM_SELF,N,N,ze,row,Mp,ierr)*
>>
>>
>> .................................................................................................................................................................................................
>>
>> But I get the following error message :
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: New nonzero at (61,124) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>> off this check
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue
>> Apr 23 21:39:26 2019
>> [0]PETSC ERROR: Configure options --with-debugging=0
>> --download-fblaslapack=1 PETSC_ARCH=arch-opt
>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 481 in
>> /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
>> [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>> /home/maahi/petsc/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: New nonzero at (124,186) caused a malloc
>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>> off this check
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue
>> Apr 23 21:39:26 2019
>> [0]PETSC ERROR: Configure options --with-debugging=0
>> --download-fblaslapack=1 PETSC_ARCH=arch-opt
>> [0]PETSC ERROR: #3 MatSetValues_SeqAIJ() line 481 in
>> /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
>> [0]PETSC ERROR: #4 MatSetValues() line 1349 in
>> /home/maahi/petsc/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>>
>> .................................................................................................................................................................................
>>
>> But instead of using 'nnz', if I put a upper bound for 'nz',  the code
>> works fine.
>>
>> Any idea what went wrong?
>>
>> Thanks,
>> Maahi Talukder
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/2d149c26/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr 23 23:51:18 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 24 Apr 2019 04:51:18 +0000
Subject: [petsc-users] Preallocation of sequential matrix
In-Reply-To: <CAMYG4GnP+YqaHC6OFfnbB5LohoxFagATx-eRZJF3Yf+et9V2Gw@mail.gmail.com>
References: <CAJOaupGtuO0NsrczzQSWdmMY57wnCrvs3FzH7s1PpF0ALp1cTg@mail.gmail.com>
	<CAMYG4GnP+YqaHC6OFfnbB5LohoxFagATx-eRZJF3Yf+et9V2Gw@mail.gmail.com>
Message-ID: <FC8F2B6C-8646-4F84-8AAF-265D90A34255@anl.gov>


  For a very small problem that produces the error add code that prints the values of row() to the screen and run with the PETSc options

 -mat_new_nonzero_allocation_err false -mat_view 

This will allow the matrix assembly to proceed and display the matrix, you can then compare the values in row() with the actual lengths of the matrix rows to get an idea of what is going wrong.  For example, for this size problem look at row 61 and compare row(61) with the length of that row in the matrix.


   Barry


> On Apr 23, 2019, at 9:05 PM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Tue, Apr 23, 2019 at 9:57 PM Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Dear All,
> 
> 
> I am trying to preallocate the no of nonzeros in my matrix using the parameter 'nnz'. Here 'row' is the parameter 'nnz'. The part of the code that does that is the following- 
> 
> ..........................................................................................................................................................................................
> Do j = 2,xmax-1
> 
> Do i = 2,ymax-1
> 
> a = (ymax-2)*(j-2)+i-1-1
> 
> If(j.eq.2 .and. i .ge. 3 .and. i .le. (ymax-2))then
> row(a) = 6
> 
> else if (j.eq.(xmax-1) .and. i.ge.3 .and. i .le. (ymax-2)) then
> row(a) = 6
> 
> else if(i.eq.2 .and. j.ge.3 .and. j.le.(xmax-2))then
> row(a) = 6
> 
> else if(i.eq.(ymax-1) .and. j.ge.3 .and. j.le.(xmax-2)) then
> row(a) = 6
> 
> else if(i.eq.2 .and. j.eq.2) then
> row(a) = 4
> 
> else if (i.eq.2 .and. j .eq. (xmax-1)) then
> row(a)= 4
> 
> else if (i.eq.(ymax-1) .and. j .eq. 2) then
> row(a) = 4
> 
> else if (i .eq. (ymax-1) .and. j .eq. (xmax-1)) then
> row(a) = 4
> 
> else
> row(a) = 9
> 
> end if
> 
> 
> end do 
> 
> end do 
> 
> 
> call MatCreateSeqAIJ(PETSC_COMM_SELF,N,N,ze,row,Mp,ierr)
> 
> .................................................................................................................................................................................................
> 
> But I get the following error message :
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (61,124) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr 23 21:39:26 2019
> [0]PETSC ERROR: Configure options --with-debugging=0 --download-fblaslapack=1 PETSC_ARCH=arch-opt
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 481 in /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1349 in /home/maahi/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (124,186) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [0]PETSC ERROR: ./Test5 on a arch-opt named CB272PP-THINK1 by maahi Tue Apr 23 21:39:26 2019
> [0]PETSC ERROR: Configure options --with-debugging=0 --download-fblaslapack=1 PETSC_ARCH=arch-opt
> [0]PETSC ERROR: #3 MatSetValues_SeqAIJ() line 481 in /home/maahi/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #4 MatSetValues() line 1349 in /home/maahi/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> .................................................................................................................................................................................
> 
> But instead of using 'nnz', if I put a upper bound for 'nz',  the code works fine.
> 
> Any idea what went wrong?
> 
> I am guessing you are not converting to the global row index the same as PETSc.
> 
>    Matt
>  
> Thanks,
> Maahi Talukder
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From eda.oktay at metu.edu.tr  Wed Apr 24 05:03:38 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Wed, 24 Apr 2019 13:03:38 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
Message-ID: <CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>

Dear Stefano,

Thank you for answering. When I used MatSetSizes, I got an error since the
matrix is read from outside, but from the error I understood that actually
the local sizes are 2127 and 2126, so I misunderstood the problem. I am
sorry for my mistake.

However, I still cannot understand where is the error. Because both
communicators and local sizes of IS and the matrix are the same. I still
get the same error in MatPermute:

[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Index -1081207334 is out of range
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
11:22:15 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
/home/edaoktay/petsc-3.10.3/include/petscis.h
[0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
/home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #3 MatPermute() line 4997 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 main() line 361 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c

This is the part of my program:

  ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
  ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
  ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
%g\n",kr);CHKERRQ(ierr);

  /* sort second smallest eigenvector */

  ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
  ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
  for (i=0; i<siz;i++) idx[i] = i;

  VecScatter ctx;
  Vec vout;
  VecScatterCreateToAll(vr,&ctx,&vout);
  VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
  VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
  VecScatterDestroy(&ctx);

  PetscScalar *avr;
  ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
  ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);

  /*Select out a piece of the resulting indices idx on each process; for
example with two processes I think rank = 0 would get the first half of the
idx and rank = 1 would get the second half.*/

  PetscMPIInt rank,size;
  MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
  MPI_Comm_size(PETSC_COMM_WORLD, &size);

  PetscInt mod;
  mod = siz % size;

  PetscInt *idxx,ss;
  ss = (siz-mod)/size;

  if (mod != 0){
    if (rank<mod){
        PetscMalloc1(ss+1,&idxx);
    } else{
        PetscMalloc1(ss,&idxx);
    }
  } else{
      PetscMalloc1(ss,&idxx);
  }


    j =0;
    for (i=rank*ss; i<(rank+1)*ss; i++) {
      idxx[j] = idx[i];
     //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
      j++;

  if (mod != 0){
    if (rank<mod){
        idxx[ss+1] = idx[ss*size+rank+1];
    }
  }

  /*Permute matrix L (spy(A(p1,p1))*/

    if (mod != 0){
        if (rank<mod){
            ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
        } else{
            ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
        }

    }else {
        ierr =
ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
    }

  ierr = ISSetPermutation(is);CHKERRQ(ierr);
  ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);

I printed IS and idxx, IS is a permutation and numbers seem to be correct.
The program also works when mod==0.

Thanks,

Eda

Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
tarihinde ?unu yazd?:

> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
> default local sizes should be 2127 and 2126.
>
> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> ha scritto:
>
>> Thank you for your answers. I figured out that for a matrix of size
>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>> wrote the program such that the local sizes of IS 2127 and 2126.
>>
>> Is local size of Mat being 2127 on both processors correct? If it is,
>> then I will change the local size of IS but then it will exceed the global
>> size of Mat. Isn't this also a problem?
>>
>> Thanks a lot,
>>
>> Eda
>>
>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
>> ?unu yazd?:
>>
>>>
>>>   Suggest printing out the IS with ISView after it is created and
>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>> each process matches the number of rows on that process.
>>>
>>>   Note from the manual page: The index sets should be on the same
>>> communicator as Mat and have the same local sizes.
>>>
>>> Barry
>>>
>>>
>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >
>>> > Hello again,
>>> >
>>> > I solved the problem for even numbered sized matrices. However when
>>> the matrix size is odd, then number of elements in each index set at each
>>> processor are different. (For example, for size 4253*4253 and 2 processors,
>>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>>> I think this is why, MatPermute again gives the same "Argument out of
>>> range" error. Index sets look like correct but I still did not get why I
>>> get this error.
>>> >
>>> > This is the part of my program:
>>> >
>>> >   PetscMPIInt rank,size;
>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>> >
>>> >   PetscInt mod;
>>> >   mod = siz % size;
>>> >
>>> >   PetscInt *idxx,ss;
>>> >   ss = (siz-mod)/size;
>>>
>>> >
>>> >   if (mod != 0){
>>> >     if (rank<mod){
>>> >         PetscMalloc1(ss+1,&idxx);
>>>
>>> >     } else{
>>> >         PetscMalloc1(ss,&idxx);
>>> >     }
>>> >   }
>>> >
>>> >   if (rank != size-1) {
>>> >     j =0;
>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>> >       idxx[j] = idx[i];
>>> >       j++;
>>> >     }
>>> >
>>> >   } else {
>>> >
>>> >     j =0;
>>> >     for (i=rank*ss; i<siz; i++) {
>>> >       idxx[j] = idx[i];
>>> >       j++;
>>> >     }
>>> >
>>> >   }
>>> >
>>> >   if (mod != 0){
>>> >     if (rank<mod){
>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>> >     }
>>> >   }
>>> >
>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>> >
>>> >     if (mod != 0){
>>> >     if (rank<mod){
>>> >         ierr =
>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>> >     } else{
>>> >         ierr =
>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>> >     }
>>> >   }
>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>> >
>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>> >
>>> > And I get the following error even if I use MatSetOption :
>>> >
>>> > [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > [0]PETSC ERROR: Argument out of range
>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>> turn off this check
>>> > [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
>>> 11:10:59 2019
>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>> --download-metis --download-parmetis --download-superlu_dist
>>> --download-slepc --download-mpich
>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> > [0]PETSC ERROR: #5 main() line 352 in
>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>> > [0]PETSC ERROR: -f
>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>> > [0]PETSC ERROR: -unweighted
>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>> entire error message to petsc-maint at mcs.anl.gov----------
>>> >
>>> > Thanks!
>>> >
>>> > Eda
>>> >
>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
>>> ?unu yazd?:
>>> > I attached whole program I wrote where the problem is in line 285. One
>>> of the matrices I used was airfoil1_binary, included in the folder. Also, I
>>> included makefile. Is that what you want?
>>> >
>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41 tarihinde
>>> ?unu yazd?:
>>> > That should not happen. Can you send in a small example that we can
>>> debug.
>>> >
>>> >   Thanks,
>>> >
>>> >     Matt
>>> >
>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> > Hello,
>>> >
>>> > I am trying to permute a vector A using following lines:
>>> >
>>> > ierr =
>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>> >   ierr =
>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>> >
>>> > However, in MatPermute line, I get the following error even if I used
>>> MatSetOption before this line:
>>> >
>>> > [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > [0]PETSC ERROR: Argument out of range
>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>> turn off this check
>>> > [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>> --download-metis --download-parmetis --download-superlu_dist
>>> --download-slepc --download-mpich
>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> > [0]PETSC ERROR: #6 main() line 285 in
>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>> > [0]PETSC ERROR: -f
>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>> > [0]PETSC ERROR: -weighted
>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>> entire error message to petsc-maint at mcs.anl.gov----------
>>> >
>>> > I'll be glad if you can help me.
>>> >
>>> > Thanks!
>>> >
>>> > Eda
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/57f8f31f/attachment-0001.html>

From stefano.zampini at gmail.com  Wed Apr 24 05:14:53 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 24 Apr 2019 13:14:53 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
Message-ID: <CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>

I don't understand your code. However, I suspect we have a bug here
https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
The code assumes the number of leaves is A->rmap->n (the row local size of
the matrix). Shouldn't this be the local size of the IS (rowp)? Barry?

Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <eda.oktay at metu.edu.tr>
ha scritto:

> Dear Stefano,
>
> Thank you for answering. When I used MatSetSizes, I got an error since the
> matrix is read from outside, but from the error I understood that actually
> the local sizes are 2127 and 2126, so I misunderstood the problem. I am
> sorry for my mistake.
>
> However, I still cannot understand where is the error. Because both
> communicators and local sizes of IS and the matrix are the same. I still
> get the same error in MatPermute:
>
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Index -1081207334 is out of range
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
> 11:22:15 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
> /home/edaoktay/petsc-3.10.3/include/petscis.h
> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #3 MatPermute() line 4997 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 main() line 361 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>
> This is the part of my program:
>
>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
> %g\n",kr);CHKERRQ(ierr);
>
>   /* sort second smallest eigenvector */
>
>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>   for (i=0; i<siz;i++) idx[i] = i;
>
>   VecScatter ctx;
>   Vec vout;
>   VecScatterCreateToAll(vr,&ctx,&vout);
>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>   VecScatterDestroy(&ctx);
>
>   PetscScalar *avr;
>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>
>   /*Select out a piece of the resulting indices idx on each process; for
> example with two processes I think rank = 0 would get the first half of the
> idx and rank = 1 would get the second half.*/
>
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>
>   PetscInt mod;
>   mod = siz % size;
>
>   PetscInt *idxx,ss;
>   ss = (siz-mod)/size;
>
>   if (mod != 0){
>     if (rank<mod){
>         PetscMalloc1(ss+1,&idxx);
>     } else{
>         PetscMalloc1(ss,&idxx);
>     }
>   } else{
>       PetscMalloc1(ss,&idxx);
>   }
>
>
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>       j++;
>
>   if (mod != 0){
>     if (rank<mod){
>         idxx[ss+1] = idx[ss*size+rank+1];
>     }
>   }
>
>   /*Permute matrix L (spy(A(p1,p1))*/
>
>     if (mod != 0){
>         if (rank<mod){
>             ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         } else{
>             ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         }
>
>     }else {
>         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     }
>
>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>
> I printed IS and idxx, IS is a permutation and numbers seem to be correct.
> The program also works when mod==0.
>
> Thanks,
>
> Eda
>
> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
> tarihinde ?unu yazd?:
>
>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>> default local sizes should be 2127 and 2126.
>>
>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> ha scritto:
>>
>>> Thank you for your answers. I figured out that for a matrix of size
>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>
>>> Is local size of Mat being 2127 on both processors correct? If it is,
>>> then I will change the local size of IS but then it will exceed the global
>>> size of Mat. Isn't this also a problem?
>>>
>>> Thanks a lot,
>>>
>>> Eda
>>>
>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
>>> ?unu yazd?:
>>>
>>>>
>>>>   Suggest printing out the IS with ISView after it is created and
>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>> each process matches the number of rows on that process.
>>>>
>>>>   Note from the manual page: The index sets should be on the same
>>>> communicator as Mat and have the same local sizes.
>>>>
>>>> Barry
>>>>
>>>>
>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> >
>>>> > Hello again,
>>>> >
>>>> > I solved the problem for even numbered sized matrices. However when
>>>> the matrix size is odd, then number of elements in each index set at each
>>>> processor are different. (For example, for size 4253*4253 and 2 processors,
>>>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>>>> I think this is why, MatPermute again gives the same "Argument out of
>>>> range" error. Index sets look like correct but I still did not get why I
>>>> get this error.
>>>> >
>>>> > This is the part of my program:
>>>> >
>>>> >   PetscMPIInt rank,size;
>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>> >
>>>> >   PetscInt mod;
>>>> >   mod = siz % size;
>>>> >
>>>> >   PetscInt *idxx,ss;
>>>> >   ss = (siz-mod)/size;
>>>>
>>>> >
>>>> >   if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>
>>>> >     } else{
>>>> >         PetscMalloc1(ss,&idxx);
>>>> >     }
>>>> >   }
>>>> >
>>>> >   if (rank != size-1) {
>>>> >     j =0;
>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>> >       idxx[j] = idx[i];
>>>> >       j++;
>>>> >     }
>>>> >
>>>> >   } else {
>>>> >
>>>> >     j =0;
>>>> >     for (i=rank*ss; i<siz; i++) {
>>>> >       idxx[j] = idx[i];
>>>> >       j++;
>>>> >     }
>>>> >
>>>> >   }
>>>> >
>>>> >   if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>> >     }
>>>> >   }
>>>> >
>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>> >
>>>> >     if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >     } else{
>>>> >         ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >     }
>>>> >   }
>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>> >
>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>> >
>>>> > And I get the following error even if I use MatSetOption :
>>>> >
>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> > [0]PETSC ERROR: Argument out of range
>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>> turn off this check
>>>> > [0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>> shooting.
>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
>>>> 11:10:59 2019
>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>> > [0]PETSC ERROR: -f
>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>> > [0]PETSC ERROR: -unweighted
>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Eda
>>>> >
>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
>>>> ?unu yazd?:
>>>> > I attached whole program I wrote where the problem is in line 285.
>>>> One of the matrices I used was airfoil1_binary, included in the folder.
>>>> Also, I included makefile. Is that what you want?
>>>> >
>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>> tarihinde ?unu yazd?:
>>>> > That should not happen. Can you send in a small example that we can
>>>> debug.
>>>> >
>>>> >   Thanks,
>>>> >
>>>> >     Matt
>>>> >
>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> > Hello,
>>>> >
>>>> > I am trying to permute a vector A using following lines:
>>>> >
>>>> > ierr =
>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>> >   ierr =
>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>> >
>>>> > However, in MatPermute line, I get the following error even if I used
>>>> MatSetOption before this line:
>>>> >
>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> > [0]PETSC ERROR: Argument out of range
>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>> turn off this check
>>>> > [0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>> shooting.
>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>> > [0]PETSC ERROR: -f
>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>> > [0]PETSC ERROR: -weighted
>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>> >
>>>> > I'll be glad if you can help me.
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Eda
>>>> >
>>>> >
>>>> > --
>>>> > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > -- Norbert Wiener
>>>> >
>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>
>>>>
>>
>> --
>> Stefano
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/ccdfcde4/attachment.html>

From eda.oktay at metu.edu.tr  Wed Apr 24 05:21:33 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Wed, 24 Apr 2019 13:21:33 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
	<CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
Message-ID: <CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>

Since I am using square matrix, isn't the row local size of the matrix
equal to the local size of IS? They are both 2127 and 2126.

Stefano Zampini <stefano.zampini at gmail.com>, 24 Nis 2019 ?ar, 13:15
tarihinde ?unu yazd?:

> I don't understand your code. However, I suspect we have a bug here
> https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
> The code assumes the number of leaves is A->rmap->n (the row local size of
> the matrix). Shouldn't this be the local size of the IS (rowp)? Barry?
>
> Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <eda.oktay at metu.edu.tr>
> ha scritto:
>
>> Dear Stefano,
>>
>> Thank you for answering. When I used MatSetSizes, I got an error since
>> the matrix is read from outside, but from the error I understood that
>> actually the local sizes are 2127 and 2126, so I misunderstood the problem.
>> I am sorry for my mistake.
>>
>> However, I still cannot understand where is the error. Because both
>> communicators and local sizes of IS and the matrix are the same. I still
>> get the same error in MatPermute:
>>
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: Index -1081207334 is out of range
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
>> 11:22:15 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
>> /home/edaoktay/petsc-3.10.3/include/petscis.h
>> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> [0]PETSC ERROR: #3 MatPermute() line 4997 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #4 main() line 361 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>
>> This is the part of my program:
>>
>>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
>> %g\n",kr);CHKERRQ(ierr);
>>
>>   /* sort second smallest eigenvector */
>>
>>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>>   for (i=0; i<siz;i++) idx[i] = i;
>>
>>   VecScatter ctx;
>>   Vec vout;
>>   VecScatterCreateToAll(vr,&ctx,&vout);
>>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>   VecScatterDestroy(&ctx);
>>
>>   PetscScalar *avr;
>>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>>
>>   /*Select out a piece of the resulting indices idx on each process; for
>> example with two processes I think rank = 0 would get the first half of the
>> idx and rank = 1 would get the second half.*/
>>
>>   PetscMPIInt rank,size;
>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>
>>   PetscInt mod;
>>   mod = siz % size;
>>
>>   PetscInt *idxx,ss;
>>   ss = (siz-mod)/size;
>>
>>   if (mod != 0){
>>     if (rank<mod){
>>         PetscMalloc1(ss+1,&idxx);
>>     } else{
>>         PetscMalloc1(ss,&idxx);
>>     }
>>   } else{
>>       PetscMalloc1(ss,&idxx);
>>   }
>>
>>
>>     j =0;
>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>       idxx[j] = idx[i];
>>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>>       j++;
>>
>>   if (mod != 0){
>>     if (rank<mod){
>>         idxx[ss+1] = idx[ss*size+rank+1];
>>     }
>>   }
>>
>>   /*Permute matrix L (spy(A(p1,p1))*/
>>
>>     if (mod != 0){
>>         if (rank<mod){
>>             ierr =
>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>         } else{
>>             ierr =
>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>         }
>>
>>     }else {
>>         ierr =
>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>     }
>>
>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>
>> I printed IS and idxx, IS is a permutation and numbers seem to be
>> correct. The program also works when mod==0.
>>
>> Thanks,
>>
>> Eda
>>
>> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
>> tarihinde ?unu yazd?:
>>
>>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>>> default local sizes should be 2127 and 2126.
>>>
>>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>>> petsc-users at mcs.anl.gov> ha scritto:
>>>
>>>> Thank you for your answers. I figured out that for a matrix of size
>>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>>
>>>> Is local size of Mat being 2127 on both processors correct? If it is,
>>>> then I will change the local size of IS but then it will exceed the global
>>>> size of Mat. Isn't this also a problem?
>>>>
>>>> Thanks a lot,
>>>>
>>>> Eda
>>>>
>>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
>>>> ?unu yazd?:
>>>>
>>>>>
>>>>>   Suggest printing out the IS with ISView after it is created and
>>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>>> each process matches the number of rows on that process.
>>>>>
>>>>>   Note from the manual page: The index sets should be on the same
>>>>> communicator as Mat and have the same local sizes.
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> >
>>>>> > Hello again,
>>>>> >
>>>>> > I solved the problem for even numbered sized matrices. However when
>>>>> the matrix size is odd, then number of elements in each index set at each
>>>>> processor are different. (For example, for size 4253*4253 and 2 processors,
>>>>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>>>>> I think this is why, MatPermute again gives the same "Argument out of
>>>>> range" error. Index sets look like correct but I still did not get why I
>>>>> get this error.
>>>>> >
>>>>> > This is the part of my program:
>>>>> >
>>>>> >   PetscMPIInt rank,size;
>>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>> >
>>>>> >   PetscInt mod;
>>>>> >   mod = siz % size;
>>>>> >
>>>>> >   PetscInt *idxx,ss;
>>>>> >   ss = (siz-mod)/size;
>>>>>
>>>>> >
>>>>> >   if (mod != 0){
>>>>> >     if (rank<mod){
>>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>>
>>>>> >     } else{
>>>>> >         PetscMalloc1(ss,&idxx);
>>>>> >     }
>>>>> >   }
>>>>> >
>>>>> >   if (rank != size-1) {
>>>>> >     j =0;
>>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>> >       idxx[j] = idx[i];
>>>>> >       j++;
>>>>> >     }
>>>>> >
>>>>> >   } else {
>>>>> >
>>>>> >     j =0;
>>>>> >     for (i=rank*ss; i<siz; i++) {
>>>>> >       idxx[j] = idx[i];
>>>>> >       j++;
>>>>> >     }
>>>>> >
>>>>> >   }
>>>>> >
>>>>> >   if (mod != 0){
>>>>> >     if (rank<mod){
>>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>>> >     }
>>>>> >   }
>>>>> >
>>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>>> >
>>>>> >     if (mod != 0){
>>>>> >     if (rank<mod){
>>>>> >         ierr =
>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>> >     } else{
>>>>> >         ierr =
>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>> >     }
>>>>> >   }
>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>> >
>>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>> >
>>>>> > And I get the following error even if I use MatSetOption :
>>>>> >
>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > [0]PETSC ERROR: Argument out of range
>>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>>> turn off this check
>>>>> > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
>>>>> 11:10:59 2019
>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>> --download-slepc --download-mpich
>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>> > [0]PETSC ERROR: -f
>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>> > [0]PETSC ERROR: -unweighted
>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>> >
>>>>> > Thanks!
>>>>> >
>>>>> > Eda
>>>>> >
>>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
>>>>> ?unu yazd?:
>>>>> > I attached whole program I wrote where the problem is in line 285.
>>>>> One of the matrices I used was airfoil1_binary, included in the folder.
>>>>> Also, I included makefile. Is that what you want?
>>>>> >
>>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>>> tarihinde ?unu yazd?:
>>>>> > That should not happen. Can you send in a small example that we can
>>>>> debug.
>>>>> >
>>>>> >   Thanks,
>>>>> >
>>>>> >     Matt
>>>>> >
>>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> > Hello,
>>>>> >
>>>>> > I am trying to permute a vector A using following lines:
>>>>> >
>>>>> > ierr =
>>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>> >   ierr =
>>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>> >
>>>>> > However, in MatPermute line, I get the following error even if I
>>>>> used MatSetOption before this line:
>>>>> >
>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> > [0]PETSC ERROR: Argument out of range
>>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>>> turn off this check
>>>>> > [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>> --download-slepc --download-mpich
>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>> > [0]PETSC ERROR: -f
>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>> > [0]PETSC ERROR: -weighted
>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>> >
>>>>> > I'll be glad if you can help me.
>>>>> >
>>>>> > Thanks!
>>>>> >
>>>>> > Eda
>>>>> >
>>>>> >
>>>>> > --
>>>>> > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > -- Norbert Wiener
>>>>> >
>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>
>>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/7d879f06/attachment-0001.html>

From stefano.zampini at gmail.com  Wed Apr 24 05:34:34 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 24 Apr 2019 13:34:34 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
	<CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
	<CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>
Message-ID: <CAGPUisinZiYToZab7hjwZt2+hoieKL=0tLbfgPwHXKR8ZNG7GA@mail.gmail.com>

Dump the index sets and the matrix in binary and send them

Il giorno mer 24 apr 2019 alle ore 13:21 Eda Oktay <eda.oktay at metu.edu.tr>
ha scritto:

> Since I am using square matrix, isn't the row local size of the matrix
> equal to the local size of IS? They are both 2127 and 2126.
>
> Stefano Zampini <stefano.zampini at gmail.com>, 24 Nis 2019 ?ar, 13:15
> tarihinde ?unu yazd?:
>
>> I don't understand your code. However, I suspect we have a bug here
>> https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
>> The code assumes the number of leaves is A->rmap->n (the row local size
>> of the matrix). Shouldn't this be the local size of the IS (rowp)? Barry?
>>
>> Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <eda.oktay at metu.edu.tr>
>> ha scritto:
>>
>>> Dear Stefano,
>>>
>>> Thank you for answering. When I used MatSetSizes, I got an error since
>>> the matrix is read from outside, but from the error I understood that
>>> actually the local sizes are 2127 and 2126, so I misunderstood the problem.
>>> I am sorry for my mistake.
>>>
>>> However, I still cannot understand where is the error. Because both
>>> communicators and local sizes of IS and the matrix are the same. I still
>>> get the same error in MatPermute:
>>>
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: Index -1081207334 is out of range
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on
>>> a arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
>>> 11:22:15 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>> --download-metis --download-parmetis --download-superlu_dist
>>> --download-slepc --download-mpich
>>> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
>>> /home/edaoktay/petsc-3.10.3/include/petscis.h
>>> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #3 MatPermute() line 4997 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #4 main() line 361 in
>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>
>>> This is the part of my program:
>>>
>>>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>>>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>>>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
>>> %g\n",kr);CHKERRQ(ierr);
>>>
>>>   /* sort second smallest eigenvector */
>>>
>>>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>>>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>>>   for (i=0; i<siz;i++) idx[i] = i;
>>>
>>>   VecScatter ctx;
>>>   Vec vout;
>>>   VecScatterCreateToAll(vr,&ctx,&vout);
>>>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>   VecScatterDestroy(&ctx);
>>>
>>>   PetscScalar *avr;
>>>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>>>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>>>
>>>   /*Select out a piece of the resulting indices idx on each process; for
>>> example with two processes I think rank = 0 would get the first half of the
>>> idx and rank = 1 would get the second half.*/
>>>
>>>   PetscMPIInt rank,size;
>>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>
>>>   PetscInt mod;
>>>   mod = siz % size;
>>>
>>>   PetscInt *idxx,ss;
>>>   ss = (siz-mod)/size;
>>>
>>>   if (mod != 0){
>>>     if (rank<mod){
>>>         PetscMalloc1(ss+1,&idxx);
>>>     } else{
>>>         PetscMalloc1(ss,&idxx);
>>>     }
>>>   } else{
>>>       PetscMalloc1(ss,&idxx);
>>>   }
>>>
>>>
>>>     j =0;
>>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>       idxx[j] = idx[i];
>>>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>>>       j++;
>>>
>>>   if (mod != 0){
>>>     if (rank<mod){
>>>         idxx[ss+1] = idx[ss*size+rank+1];
>>>     }
>>>   }
>>>
>>>   /*Permute matrix L (spy(A(p1,p1))*/
>>>
>>>     if (mod != 0){
>>>         if (rank<mod){
>>>             ierr =
>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>         } else{
>>>             ierr =
>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>         }
>>>
>>>     }else {
>>>         ierr =
>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>     }
>>>
>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>
>>> I printed IS and idxx, IS is a permutation and numbers seem to be
>>> correct. The program also works when mod==0.
>>>
>>> Thanks,
>>>
>>> Eda
>>>
>>> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
>>> tarihinde ?unu yazd?:
>>>
>>>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>>>> default local sizes should be 2127 and 2126.
>>>>
>>>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> ha scritto:
>>>>
>>>>> Thank you for your answers. I figured out that for a matrix of size
>>>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>>>
>>>>> Is local size of Mat being 2127 on both processors correct? If it is,
>>>>> then I will change the local size of IS but then it will exceed the global
>>>>> size of Mat. Isn't this also a problem?
>>>>>
>>>>> Thanks a lot,
>>>>>
>>>>> Eda
>>>>>
>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
>>>>> ?unu yazd?:
>>>>>
>>>>>>
>>>>>>   Suggest printing out the IS with ISView after it is created and
>>>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>>>> each process matches the number of rows on that process.
>>>>>>
>>>>>>   Note from the manual page: The index sets should be on the same
>>>>>> communicator as Mat and have the same local sizes.
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>> >
>>>>>> > Hello again,
>>>>>> >
>>>>>> > I solved the problem for even numbered sized matrices. However when
>>>>>> the matrix size is odd, then number of elements in each index set at each
>>>>>> processor are different. (For example, for size 4253*4253 and 2 processors,
>>>>>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>>>>>> I think this is why, MatPermute again gives the same "Argument out of
>>>>>> range" error. Index sets look like correct but I still did not get why I
>>>>>> get this error.
>>>>>> >
>>>>>> > This is the part of my program:
>>>>>> >
>>>>>> >   PetscMPIInt rank,size;
>>>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>> >
>>>>>> >   PetscInt mod;
>>>>>> >   mod = siz % size;
>>>>>> >
>>>>>> >   PetscInt *idxx,ss;
>>>>>> >   ss = (siz-mod)/size;
>>>>>>
>>>>>> >
>>>>>> >   if (mod != 0){
>>>>>> >     if (rank<mod){
>>>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>>>
>>>>>> >     } else{
>>>>>> >         PetscMalloc1(ss,&idxx);
>>>>>> >     }
>>>>>> >   }
>>>>>> >
>>>>>> >   if (rank != size-1) {
>>>>>> >     j =0;
>>>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>> >       idxx[j] = idx[i];
>>>>>> >       j++;
>>>>>> >     }
>>>>>> >
>>>>>> >   } else {
>>>>>> >
>>>>>> >     j =0;
>>>>>> >     for (i=rank*ss; i<siz; i++) {
>>>>>> >       idxx[j] = idx[i];
>>>>>> >       j++;
>>>>>> >     }
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >   if (mod != 0){
>>>>>> >     if (rank<mod){
>>>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>> >     }
>>>>>> >   }
>>>>>> >
>>>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>> >
>>>>>> >     if (mod != 0){
>>>>>> >     if (rank<mod){
>>>>>> >         ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>> >     } else{
>>>>>> >         ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>> >     }
>>>>>> >   }
>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>> >
>>>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>> >
>>>>>> > And I get the following error even if I use MatSetOption :
>>>>>> >
>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>>>> turn off this check
>>>>>> > [0]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr
>>>>>> 8 11:10:59 2019
>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>> --download-slepc --download-mpich
>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>> > [0]PETSC ERROR: -f
>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>> > [0]PETSC ERROR: -unweighted
>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>> >
>>>>>> > Thanks!
>>>>>> >
>>>>>> > Eda
>>>>>> >
>>>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53
>>>>>> tarihinde ?unu yazd?:
>>>>>> > I attached whole program I wrote where the problem is in line 285.
>>>>>> One of the matrices I used was airfoil1_binary, included in the folder.
>>>>>> Also, I included makefile. Is that what you want?
>>>>>> >
>>>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>>>> tarihinde ?unu yazd?:
>>>>>> > That should not happen. Can you send in a small example that we can
>>>>>> debug.
>>>>>> >
>>>>>> >   Thanks,
>>>>>> >
>>>>>> >     Matt
>>>>>> >
>>>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>> > Hello,
>>>>>> >
>>>>>> > I am trying to permute a vector A using following lines:
>>>>>> >
>>>>>> > ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>>> >   ierr =
>>>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>>> >
>>>>>> > However, in MatPermute line, I get the following error even if I
>>>>>> used MatSetOption before this line:
>>>>>> >
>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>>>> turn off this check
>>>>>> > [0]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>> arch-linux2-c-debug named 1232.wls.metu.edu.tr by edaoktay Mon Mar
>>>>>> 25 12:15:14 2019
>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>> --download-slepc --download-mpich
>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>> > [0]PETSC ERROR: -f
>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>> > [0]PETSC ERROR: -weighted
>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>> >
>>>>>> > I'll be glad if you can help me.
>>>>>> >
>>>>>> > Thanks!
>>>>>> >
>>>>>> > Eda
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > -- Norbert Wiener
>>>>>> >
>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Stefano
>>>>
>>>
>>
>> --
>> Stefano
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/a540f622/attachment-0001.html>

From knepley at gmail.com  Wed Apr 24 05:34:58 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Apr 2019 06:34:58 -0400
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
Message-ID: <CAMYG4G=9BVgVnN2hfqwv5rctAcFGGw8DkkGQDN6hcdknA+BsGQ@mail.gmail.com>

On Wed, Apr 24, 2019 at 6:06 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear Stefano,
>
> Thank you for answering. When I used MatSetSizes, I got an error since the
> matrix is read from outside, but from the error I understood that actually
> the local sizes are 2127 and 2126, so I misunderstood the problem. I am
> sorry for my mistake.
>
> However, I still cannot understand where is the error. Because both
> communicators and local sizes of IS and the matrix are the same. I still
> get the same error in MatPermute:
>
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Index -1081207334 is out of range
>

It looks like you are sticking an invalid value into the IS, probably
because of a mixup on sizes in parallel.

  Thanks,

     Matt


> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
> 11:22:15 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
> /home/edaoktay/petsc-3.10.3/include/petscis.h
> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #3 MatPermute() line 4997 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 main() line 361 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>
> This is the part of my program:
>
>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
> %g\n",kr);CHKERRQ(ierr);
>
>   /* sort second smallest eigenvector */
>
>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>   for (i=0; i<siz;i++) idx[i] = i;
>
>   VecScatter ctx;
>   Vec vout;
>   VecScatterCreateToAll(vr,&ctx,&vout);
>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>   VecScatterDestroy(&ctx);
>
>   PetscScalar *avr;
>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>
>   /*Select out a piece of the resulting indices idx on each process; for
> example with two processes I think rank = 0 would get the first half of the
> idx and rank = 1 would get the second half.*/
>
>   PetscMPIInt rank,size;
>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>
>   PetscInt mod;
>   mod = siz % size;
>
>   PetscInt *idxx,ss;
>   ss = (siz-mod)/size;
>
>   if (mod != 0){
>     if (rank<mod){
>         PetscMalloc1(ss+1,&idxx);
>     } else{
>         PetscMalloc1(ss,&idxx);
>     }
>   } else{
>       PetscMalloc1(ss,&idxx);
>   }
>
>
>     j =0;
>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>       idxx[j] = idx[i];
>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>       j++;
>
>   if (mod != 0){
>     if (rank<mod){
>         idxx[ss+1] = idx[ss*size+rank+1];
>     }
>   }
>
>   /*Permute matrix L (spy(A(p1,p1))*/
>
>     if (mod != 0){
>         if (rank<mod){
>             ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         } else{
>             ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>         }
>
>     }else {
>         ierr =
> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>     }
>
>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>
> I printed IS and idxx, IS is a permutation and numbers seem to be correct.
> The program also works when mod==0.
>
> Thanks,
>
> Eda
>
> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
> tarihinde ?unu yazd?:
>
>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>> default local sizes should be 2127 and 2126.
>>
>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> ha scritto:
>>
>>> Thank you for your answers. I figured out that for a matrix of size
>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>
>>> Is local size of Mat being 2127 on both processors correct? If it is,
>>> then I will change the local size of IS but then it will exceed the global
>>> size of Mat. Isn't this also a problem?
>>>
>>> Thanks a lot,
>>>
>>> Eda
>>>
>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31 tarihinde
>>> ?unu yazd?:
>>>
>>>>
>>>>   Suggest printing out the IS with ISView after it is created and
>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>> each process matches the number of rows on that process.
>>>>
>>>>   Note from the manual page: The index sets should be on the same
>>>> communicator as Mat and have the same local sizes.
>>>>
>>>> Barry
>>>>
>>>>
>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> >
>>>> > Hello again,
>>>> >
>>>> > I solved the problem for even numbered sized matrices. However when
>>>> the matrix size is odd, then number of elements in each index set at each
>>>> processor are different. (For example, for size 4253*4253 and 2 processors,
>>>> size of index set at processor 0 is 2127 where at processor 1, it is 2126)
>>>> I think this is why, MatPermute again gives the same "Argument out of
>>>> range" error. Index sets look like correct but I still did not get why I
>>>> get this error.
>>>> >
>>>> > This is the part of my program:
>>>> >
>>>> >   PetscMPIInt rank,size;
>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>> >
>>>> >   PetscInt mod;
>>>> >   mod = siz % size;
>>>> >
>>>> >   PetscInt *idxx,ss;
>>>> >   ss = (siz-mod)/size;
>>>>
>>>> >
>>>> >   if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>
>>>> >     } else{
>>>> >         PetscMalloc1(ss,&idxx);
>>>> >     }
>>>> >   }
>>>> >
>>>> >   if (rank != size-1) {
>>>> >     j =0;
>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>> >       idxx[j] = idx[i];
>>>> >       j++;
>>>> >     }
>>>> >
>>>> >   } else {
>>>> >
>>>> >     j =0;
>>>> >     for (i=rank*ss; i<siz; i++) {
>>>> >       idxx[j] = idx[i];
>>>> >       j++;
>>>> >     }
>>>> >
>>>> >   }
>>>> >
>>>> >   if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>> >     }
>>>> >   }
>>>> >
>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>> >
>>>> >     if (mod != 0){
>>>> >     if (rank<mod){
>>>> >         ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >     } else{
>>>> >         ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >     }
>>>> >   }
>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>> >
>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>> >
>>>> > And I get the following error even if I use MatSetOption :
>>>> >
>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> > [0]PETSC ERROR: Argument out of range
>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>> turn off this check
>>>> > [0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>> shooting.
>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr  8
>>>> 11:10:59 2019
>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>> > [0]PETSC ERROR: -f
>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>> > [0]PETSC ERROR: -unweighted
>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Eda
>>>> >
>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53 tarihinde
>>>> ?unu yazd?:
>>>> > I attached whole program I wrote where the problem is in line 285.
>>>> One of the matrices I used was airfoil1_binary, included in the folder.
>>>> Also, I included makefile. Is that what you want?
>>>> >
>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>> tarihinde ?unu yazd?:
>>>> > That should not happen. Can you send in a small example that we can
>>>> debug.
>>>> >
>>>> >   Thanks,
>>>> >
>>>> >     Matt
>>>> >
>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> > Hello,
>>>> >
>>>> > I am trying to permute a vector A using following lines:
>>>> >
>>>> > ierr =
>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>> >   ierr =
>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>> >
>>>> > However, in MatPermute line, I get the following error even if I used
>>>> MatSetOption before this line:
>>>> >
>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> > [0]PETSC ERROR: Argument out of range
>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>>> turn off this check
>>>> > [0]PETSC ERROR: See
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>> shooting.
>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug
>>>> named 1232.wls.metu.edu.tr by edaoktay Mon Mar 25 12:15:14 2019
>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>> > [0]PETSC ERROR: -f
>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>> > [0]PETSC ERROR: -weighted
>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>> >
>>>> > I'll be glad if you can help me.
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Eda
>>>> >
>>>> >
>>>> > --
>>>> > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > -- Norbert Wiener
>>>> >
>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>
>>>>
>>
>> --
>> Stefano
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/349664c5/attachment.html>

From knepley at gmail.com  Wed Apr 24 06:30:00 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Apr 2019 07:30:00 -0400
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CAGPUisinZiYToZab7hjwZt2+hoieKL=0tLbfgPwHXKR8ZNG7GA@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
	<CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
	<CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>
	<CAGPUisinZiYToZab7hjwZt2+hoieKL=0tLbfgPwHXKR8ZNG7GA@mail.gmail.com>
Message-ID: <CAMYG4G=1b+Gfc9i=N1+aYi2SQJgoH2tA=b3-1xVZUiF1vJO4nA@mail.gmail.com>

On Wed, Apr 24, 2019 at 6:35 AM Stefano Zampini via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dump the index sets and the matrix in binary and send them
>

First, reduce your problem size to about 10.

  Matt


> Il giorno mer 24 apr 2019 alle ore 13:21 Eda Oktay <eda.oktay at metu.edu.tr>
> ha scritto:
>
>> Since I am using square matrix, isn't the row local size of the matrix
>> equal to the local size of IS? They are both 2127 and 2126.
>>
>> Stefano Zampini <stefano.zampini at gmail.com>, 24 Nis 2019 ?ar, 13:15
>> tarihinde ?unu yazd?:
>>
>>> I don't understand your code. However, I suspect we have a bug here
>>> https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
>>> The code assumes the number of leaves is A->rmap->n (the row local size
>>> of the matrix). Shouldn't this be the local size of the IS (rowp)? Barry?
>>>
>>> Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <
>>> eda.oktay at metu.edu.tr> ha scritto:
>>>
>>>> Dear Stefano,
>>>>
>>>> Thank you for answering. When I used MatSetSizes, I got an error since
>>>> the matrix is read from outside, but from the error I understood that
>>>> actually the local sizes are 2127 and 2126, so I misunderstood the problem.
>>>> I am sorry for my mistake.
>>>>
>>>> However, I still cannot understand where is the error. Because both
>>>> communicators and local sizes of IS and the matrix are the same. I still
>>>> get the same error in MatPermute:
>>>>
>>>> [0]PETSC ERROR: Argument out of range
>>>> [0]PETSC ERROR: Index -1081207334 is out of range
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on
>>>> a arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed Apr 24
>>>> 11:22:15 2019
>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>> --download-metis --download-parmetis --download-superlu_dist
>>>> --download-slepc --download-mpich
>>>> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
>>>> /home/edaoktay/petsc-3.10.3/include/petscis.h
>>>> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>> [0]PETSC ERROR: #3 MatPermute() line 4997 in
>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>> [0]PETSC ERROR: #4 main() line 361 in
>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>
>>>> This is the part of my program:
>>>>
>>>>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>>>>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue:
>>>> %g\n",kr);CHKERRQ(ierr);
>>>>
>>>>   /* sort second smallest eigenvector */
>>>>
>>>>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>>>>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>>>>   for (i=0; i<siz;i++) idx[i] = i;
>>>>
>>>>   VecScatter ctx;
>>>>   Vec vout;
>>>>   VecScatterCreateToAll(vr,&ctx,&vout);
>>>>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>   VecScatterDestroy(&ctx);
>>>>
>>>>   PetscScalar *avr;
>>>>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>>>>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>>>>
>>>>   /*Select out a piece of the resulting indices idx on each process;
>>>> for example with two processes I think rank = 0 would get the first half of
>>>> the idx and rank = 1 would get the second half.*/
>>>>
>>>>   PetscMPIInt rank,size;
>>>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>
>>>>   PetscInt mod;
>>>>   mod = siz % size;
>>>>
>>>>   PetscInt *idxx,ss;
>>>>   ss = (siz-mod)/size;
>>>>
>>>>   if (mod != 0){
>>>>     if (rank<mod){
>>>>         PetscMalloc1(ss+1,&idxx);
>>>>
>>>>     } else{
>>>>         PetscMalloc1(ss,&idxx);
>>>>     }
>>>>   } else{
>>>>       PetscMalloc1(ss,&idxx);
>>>>   }
>>>>
>>>>
>>>>     j =0;
>>>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>       idxx[j] = idx[i];
>>>>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>>>>       j++;
>>>>
>>>>   if (mod != 0){
>>>>     if (rank<mod){
>>>>         idxx[ss+1] = idx[ss*size+rank+1];
>>>>     }
>>>>   }
>>>>
>>>>   /*Permute matrix L (spy(A(p1,p1))*/
>>>>
>>>>     if (mod != 0){
>>>>         if (rank<mod){
>>>>             ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>         } else{
>>>>             ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>         }
>>>>
>>>>     }else {
>>>>         ierr =
>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>     }
>>>>
>>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>
>>>> I printed IS and idxx, IS is a permutation and numbers seem to be
>>>> correct. The program also works when mod==0.
>>>>
>>>> Thanks,
>>>>
>>>> Eda
>>>>
>>>> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
>>>> tarihinde ?unu yazd?:
>>>>
>>>>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>>>>> default local sizes should be 2127 and 2126.
>>>>>
>>>>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>>>>> petsc-users at mcs.anl.gov> ha scritto:
>>>>>
>>>>>> Thank you for your answers. I figured out that for a matrix of size
>>>>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>>>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>>>>
>>>>>> Is local size of Mat being 2127 on both processors correct? If it is,
>>>>>> then I will change the local size of IS but then it will exceed the global
>>>>>> size of Mat. Isn't this also a problem?
>>>>>>
>>>>>> Thanks a lot,
>>>>>>
>>>>>> Eda
>>>>>>
>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31
>>>>>> tarihinde ?unu yazd?:
>>>>>>
>>>>>>>
>>>>>>>   Suggest printing out the IS with ISView after it is created and
>>>>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>>>>> each process matches the number of rows on that process.
>>>>>>>
>>>>>>>   Note from the manual page: The index sets should be on the same
>>>>>>> communicator as Mat and have the same local sizes.
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>>
>>>>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>> >
>>>>>>> > Hello again,
>>>>>>> >
>>>>>>> > I solved the problem for even numbered sized matrices. However
>>>>>>> when the matrix size is odd, then number of elements in each index set at
>>>>>>> each processor are different. (For example, for size 4253*4253 and 2
>>>>>>> processors, size of index set at processor 0 is 2127 where at processor 1,
>>>>>>> it is 2126) I think this is why, MatPermute again gives the same "Argument
>>>>>>> out of range" error. Index sets look like correct but I still did not get
>>>>>>> why I get this error.
>>>>>>> >
>>>>>>> > This is the part of my program:
>>>>>>> >
>>>>>>> >   PetscMPIInt rank,size;
>>>>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>>> >
>>>>>>> >   PetscInt mod;
>>>>>>> >   mod = siz % size;
>>>>>>> >
>>>>>>> >   PetscInt *idxx,ss;
>>>>>>> >   ss = (siz-mod)/size;
>>>>>>>
>>>>>>> >
>>>>>>> >   if (mod != 0){
>>>>>>> >     if (rank<mod){
>>>>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>>>>
>>>>>>> >     } else{
>>>>>>> >         PetscMalloc1(ss,&idxx);
>>>>>>> >     }
>>>>>>> >   }
>>>>>>> >
>>>>>>> >   if (rank != size-1) {
>>>>>>> >     j =0;
>>>>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>>> >       idxx[j] = idx[i];
>>>>>>> >       j++;
>>>>>>> >     }
>>>>>>> >
>>>>>>> >   } else {
>>>>>>> >
>>>>>>> >     j =0;
>>>>>>> >     for (i=rank*ss; i<siz; i++) {
>>>>>>> >       idxx[j] = idx[i];
>>>>>>> >       j++;
>>>>>>> >     }
>>>>>>> >
>>>>>>> >   }
>>>>>>> >
>>>>>>> >   if (mod != 0){
>>>>>>> >     if (rank<mod){
>>>>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>>> >     }
>>>>>>> >   }
>>>>>>> >
>>>>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>>> >
>>>>>>> >     if (mod != 0){
>>>>>>> >     if (rank<mod){
>>>>>>> >         ierr =
>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>> >     } else{
>>>>>>> >         ierr =
>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>> >     }
>>>>>>> >   }
>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>> >
>>>>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>>> >
>>>>>>> > And I get the following error even if I use MatSetOption :
>>>>>>> >
>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>> to turn off this check
>>>>>>> > [0]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon Apr
>>>>>>> 8 11:10:59 2019
>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>> --download-slepc --download-mpich
>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>> > [0]PETSC ERROR: -f
>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>> > [0]PETSC ERROR: -unweighted
>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>> >
>>>>>>> > Thanks!
>>>>>>> >
>>>>>>> > Eda
>>>>>>> >
>>>>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53
>>>>>>> tarihinde ?unu yazd?:
>>>>>>> > I attached whole program I wrote where the problem is in line 285.
>>>>>>> One of the matrices I used was airfoil1_binary, included in the folder.
>>>>>>> Also, I included makefile. Is that what you want?
>>>>>>> >
>>>>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>>>>> tarihinde ?unu yazd?:
>>>>>>> > That should not happen. Can you send in a small example that we
>>>>>>> can debug.
>>>>>>> >
>>>>>>> >   Thanks,
>>>>>>> >
>>>>>>> >     Matt
>>>>>>> >
>>>>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>> > Hello,
>>>>>>> >
>>>>>>> > I am trying to permute a vector A using following lines:
>>>>>>> >
>>>>>>> > ierr =
>>>>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>>>> >   ierr =
>>>>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>>>> >
>>>>>>> > However, in MatPermute line, I get the following error even if I
>>>>>>> used MatSetOption before this line:
>>>>>>> >
>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>> to turn off this check
>>>>>>> > [0]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>> arch-linux2-c-debug named 1232.wls.metu.edu.tr by edaoktay Mon Mar
>>>>>>> 25 12:15:14 2019
>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>> --download-slepc --download-mpich
>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>> > [0]PETSC ERROR: -f
>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>> > [0]PETSC ERROR: -weighted
>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>> >
>>>>>>> > I'll be glad if you can help me.
>>>>>>> >
>>>>>>> > Thanks!
>>>>>>> >
>>>>>>> > Eda
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > -- Norbert Wiener
>>>>>>> >
>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Stefano
>>>>>
>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>
> --
> Stefano
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/bd20f48a/attachment-0001.html>

From eda.oktay at metu.edu.tr  Wed Apr 24 07:24:50 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Wed, 24 Apr 2019 15:24:50 +0300
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CAMYG4G=1b+Gfc9i=N1+aYi2SQJgoH2tA=b3-1xVZUiF1vJO4nA@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
	<CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
	<CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>
	<CAGPUisinZiYToZab7hjwZt2+hoieKL=0tLbfgPwHXKR8ZNG7GA@mail.gmail.com>
	<CAMYG4G=1b+Gfc9i=N1+aYi2SQJgoH2tA=b3-1xVZUiF1vJO4nA@mail.gmail.com>
Message-ID: <CA+vu_XjjwEt6NiqS2Vo2_Bg=RLnXNCAFFGzQ4dQ2y-A8VVxR_w@mail.gmail.com>

Dear Matt,

I solved my problem by trying with 9*9 matrix as you said. I made indexing
mistake.

Thank you for answering.

Eda

Matthew Knepley <knepley at gmail.com>, 24 Nis 2019 ?ar, 14:30 tarihinde ?unu
yazd?:

> On Wed, Apr 24, 2019 at 6:35 AM Stefano Zampini via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dump the index sets and the matrix in binary and send them
>>
>
> First, reduce your problem size to about 10.
>
>   Matt
>
>
>> Il giorno mer 24 apr 2019 alle ore 13:21 Eda Oktay <eda.oktay at metu.edu.tr>
>> ha scritto:
>>
>>> Since I am using square matrix, isn't the row local size of the matrix
>>> equal to the local size of IS? They are both 2127 and 2126.
>>>
>>> Stefano Zampini <stefano.zampini at gmail.com>, 24 Nis 2019 ?ar, 13:15
>>> tarihinde ?unu yazd?:
>>>
>>>> I don't understand your code. However, I suspect we have a bug here
>>>> https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
>>>> The code assumes the number of leaves is A->rmap->n (the row local size
>>>> of the matrix). Shouldn't this be the local size of the IS (rowp)? Barry?
>>>>
>>>> Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <
>>>> eda.oktay at metu.edu.tr> ha scritto:
>>>>
>>>>> Dear Stefano,
>>>>>
>>>>> Thank you for answering. When I used MatSetSizes, I got an error since
>>>>> the matrix is read from outside, but from the error I understood that
>>>>> actually the local sizes are 2127 and 2126, so I misunderstood the problem.
>>>>> I am sorry for my mistake.
>>>>>
>>>>> However, I still cannot understand where is the error. Because both
>>>>> communicators and local sizes of IS and the matrix are the same. I still
>>>>> get the same error in MatPermute:
>>>>>
>>>>> [0]PETSC ERROR: Argument out of range
>>>>> [0]PETSC ERROR: Index -1081207334 is out of range
>>>>> [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL
>>>>> on a arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed
>>>>> Apr 24 11:22:15 2019
>>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>> --download-slepc --download-mpich
>>>>> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
>>>>> /home/edaoktay/petsc-3.10.3/include/petscis.h
>>>>> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>> [0]PETSC ERROR: #3 MatPermute() line 4997 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>> [0]PETSC ERROR: #4 main() line 361 in
>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>
>>>>> This is the part of my program:
>>>>>
>>>>>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>>>>>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest
>>>>> eigenvalue: %g\n",kr);CHKERRQ(ierr);
>>>>>
>>>>>   /* sort second smallest eigenvector */
>>>>>
>>>>>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>>>>>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>>>>>   for (i=0; i<siz;i++) idx[i] = i;
>>>>>
>>>>>   VecScatter ctx;
>>>>>   Vec vout;
>>>>>   VecScatterCreateToAll(vr,&ctx,&vout);
>>>>>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>>   VecScatterDestroy(&ctx);
>>>>>
>>>>>   PetscScalar *avr;
>>>>>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>>>>>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>>>>>
>>>>>   /*Select out a piece of the resulting indices idx on each process;
>>>>> for example with two processes I think rank = 0 would get the first half of
>>>>> the idx and rank = 1 would get the second half.*/
>>>>>
>>>>>   PetscMPIInt rank,size;
>>>>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>
>>>>>   PetscInt mod;
>>>>>   mod = siz % size;
>>>>>
>>>>>   PetscInt *idxx,ss;
>>>>>   ss = (siz-mod)/size;
>>>>>
>>>>>   if (mod != 0){
>>>>>     if (rank<mod){
>>>>>         PetscMalloc1(ss+1,&idxx);
>>>>>
>>>>>     } else{
>>>>>         PetscMalloc1(ss,&idxx);
>>>>>     }
>>>>>   } else{
>>>>>       PetscMalloc1(ss,&idxx);
>>>>>   }
>>>>>
>>>>>
>>>>>     j =0;
>>>>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>       idxx[j] = idx[i];
>>>>>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>>>>>       j++;
>>>>>
>>>>>   if (mod != 0){
>>>>>     if (rank<mod){
>>>>>         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>     }
>>>>>   }
>>>>>
>>>>>   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>
>>>>>     if (mod != 0){
>>>>>         if (rank<mod){
>>>>>             ierr =
>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>         } else{
>>>>>             ierr =
>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>         }
>>>>>
>>>>>     }else {
>>>>>         ierr =
>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>     }
>>>>>
>>>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>
>>>>> I printed IS and idxx, IS is a permutation and numbers seem to be
>>>>> correct. The program also works when mod==0.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Eda
>>>>>
>>>>> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
>>>>> tarihinde ?unu yazd?:
>>>>>
>>>>>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the
>>>>>> default local sizes should be 2127 and 2126.
>>>>>>
>>>>>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> ha scritto:
>>>>>>
>>>>>>> Thank you for your answers. I figured out that for a matrix of size
>>>>>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>>>>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>>>>>
>>>>>>> Is local size of Mat being 2127 on both processors correct? If it
>>>>>>> is, then I will change the local size of IS but then it will exceed the
>>>>>>> global size of Mat. Isn't this also a problem?
>>>>>>>
>>>>>>> Thanks a lot,
>>>>>>>
>>>>>>> Eda
>>>>>>>
>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31
>>>>>>> tarihinde ?unu yazd?:
>>>>>>>
>>>>>>>>
>>>>>>>>   Suggest printing out the IS with ISView after it is created and
>>>>>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>>>>>> each process matches the number of rows on that process.
>>>>>>>>
>>>>>>>>   Note from the manual page: The index sets should be on the same
>>>>>>>> communicator as Mat and have the same local sizes.
>>>>>>>>
>>>>>>>> Barry
>>>>>>>>
>>>>>>>>
>>>>>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>> >
>>>>>>>> > Hello again,
>>>>>>>> >
>>>>>>>> > I solved the problem for even numbered sized matrices. However
>>>>>>>> when the matrix size is odd, then number of elements in each index set at
>>>>>>>> each processor are different. (For example, for size 4253*4253 and 2
>>>>>>>> processors, size of index set at processor 0 is 2127 where at processor 1,
>>>>>>>> it is 2126) I think this is why, MatPermute again gives the same "Argument
>>>>>>>> out of range" error. Index sets look like correct but I still did not get
>>>>>>>> why I get this error.
>>>>>>>> >
>>>>>>>> > This is the part of my program:
>>>>>>>> >
>>>>>>>> >   PetscMPIInt rank,size;
>>>>>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>>>> >
>>>>>>>> >   PetscInt mod;
>>>>>>>> >   mod = siz % size;
>>>>>>>> >
>>>>>>>> >   PetscInt *idxx,ss;
>>>>>>>> >   ss = (siz-mod)/size;
>>>>>>>>
>>>>>>>> >
>>>>>>>> >   if (mod != 0){
>>>>>>>> >     if (rank<mod){
>>>>>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>>>>>
>>>>>>>> >     } else{
>>>>>>>> >         PetscMalloc1(ss,&idxx);
>>>>>>>> >     }
>>>>>>>> >   }
>>>>>>>> >
>>>>>>>> >   if (rank != size-1) {
>>>>>>>> >     j =0;
>>>>>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>>>> >       idxx[j] = idx[i];
>>>>>>>> >       j++;
>>>>>>>> >     }
>>>>>>>> >
>>>>>>>> >   } else {
>>>>>>>> >
>>>>>>>> >     j =0;
>>>>>>>> >     for (i=rank*ss; i<siz; i++) {
>>>>>>>> >       idxx[j] = idx[i];
>>>>>>>> >       j++;
>>>>>>>> >     }
>>>>>>>> >
>>>>>>>> >   }
>>>>>>>> >
>>>>>>>> >   if (mod != 0){
>>>>>>>> >     if (rank<mod){
>>>>>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>>>> >     }
>>>>>>>> >   }
>>>>>>>> >
>>>>>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>>>> >
>>>>>>>> >     if (mod != 0){
>>>>>>>> >     if (rank<mod){
>>>>>>>> >         ierr =
>>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>> >     } else{
>>>>>>>> >         ierr =
>>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>> >     }
>>>>>>>> >   }
>>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>>> >
>>>>>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>>>> >
>>>>>>>> > And I get the following error even if I use MatSetOption :
>>>>>>>> >
>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>>> to turn off this check
>>>>>>>> > [0]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon
>>>>>>>> Apr  8 11:10:59 2019
>>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>>> --download-slepc --download-mpich
>>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>>> > [0]PETSC ERROR: -f
>>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>>> > [0]PETSC ERROR: -unweighted
>>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>>> >
>>>>>>>> > Thanks!
>>>>>>>> >
>>>>>>>> > Eda
>>>>>>>> >
>>>>>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53
>>>>>>>> tarihinde ?unu yazd?:
>>>>>>>> > I attached whole program I wrote where the problem is in line
>>>>>>>> 285. One of the matrices I used was airfoil1_binary, included in the
>>>>>>>> folder. Also, I included makefile. Is that what you want?
>>>>>>>> >
>>>>>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>>>>>> tarihinde ?unu yazd?:
>>>>>>>> > That should not happen. Can you send in a small example that we
>>>>>>>> can debug.
>>>>>>>> >
>>>>>>>> >   Thanks,
>>>>>>>> >
>>>>>>>> >     Matt
>>>>>>>> >
>>>>>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>> > Hello,
>>>>>>>> >
>>>>>>>> > I am trying to permute a vector A using following lines:
>>>>>>>> >
>>>>>>>> > ierr =
>>>>>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>>>>> >   ierr =
>>>>>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>>>>> >
>>>>>>>> > However, in MatPermute line, I get the following error even if I
>>>>>>>> used MatSetOption before this line:
>>>>>>>> >
>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>>> to turn off this check
>>>>>>>> > [0]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>>> arch-linux2-c-debug named 1232.wls.metu.edu.tr by edaoktay Mon Mar
>>>>>>>> 25 12:15:14 2019
>>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>>> --download-slepc --download-mpich
>>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>>> > [0]PETSC ERROR: -f
>>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>>> > [0]PETSC ERROR: -weighted
>>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>>> >
>>>>>>>> > I'll be glad if you can help me.
>>>>>>>> >
>>>>>>>> > Thanks!
>>>>>>>> >
>>>>>>>> > Eda
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Stefano
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Stefano
>>>>
>>>
>>
>> --
>> Stefano
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/253f9951/attachment-0001.html>

From knepley at gmail.com  Wed Apr 24 07:26:00 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Apr 2019 08:26:00 -0400
Subject: [petsc-users] Argument out of range error in MatPermute
In-Reply-To: <CA+vu_XjjwEt6NiqS2Vo2_Bg=RLnXNCAFFGzQ4dQ2y-A8VVxR_w@mail.gmail.com>
References: <CA+vu_XhrSnniOFrduYP8YfmGSvRRDZncDm=1myHAddbn=v3Q_g@mail.gmail.com>
	<CAMYG4GmBOwwrBTfYSmLnYS_EWOA+Op=Z5y7i54RvCXeCw0nDjg@mail.gmail.com>
	<CA+vu_XhD9d0gXAbowk4MLpzsB-zu4FieNw=uH3e07_-u4T0ANQ@mail.gmail.com>
	<CA+vu_XjMnRqe+AyDiFqtOn8yaQVKoY2QV3XpDRCHxdmn2JvWOw@mail.gmail.com>
	<DD0EE3AF-0F78-417F-82C8-6900D51C9C96@anl.gov>
	<CA+vu_XibBPFtw6ewacHktFMN8TTCJMbNdxfoYJpWDh6CmYCD7Q@mail.gmail.com>
	<CAGPUisgS24VmwqL4sfPZH5QHtzCr8ohFq5jcbLh7SZVZ3jJLWQ@mail.gmail.com>
	<CA+vu_XhY+Dr_-Sm7-0JKVORf_exXnFbf_4LbSTAe=3G7GB6wbQ@mail.gmail.com>
	<CAGPUisiWwXkpEGsmkiSJLHG8_R8iJyA3sD+9oSyUaetfv2+6=w@mail.gmail.com>
	<CA+vu_Xi33tBtorKWT0zrQKXHm7JQVq85ujCiedM+MCP51rJE8Q@mail.gmail.com>
	<CAGPUisinZiYToZab7hjwZt2+hoieKL=0tLbfgPwHXKR8ZNG7GA@mail.gmail.com>
	<CAMYG4G=1b+Gfc9i=N1+aYi2SQJgoH2tA=b3-1xVZUiF1vJO4nA@mail.gmail.com>
	<CA+vu_XjjwEt6NiqS2Vo2_Bg=RLnXNCAFFGzQ4dQ2y-A8VVxR_w@mail.gmail.com>
Message-ID: <CAMYG4GmDzPdToADDzoBSaVg7uO8OJMp4=g9n-ZaS+T+dv7Ph=g@mail.gmail.com>

On Wed, Apr 24, 2019 at 8:25 AM Eda Oktay <eda.oktay at metu.edu.tr> wrote:

> Dear Matt,
>
> I solved my problem by trying with 9*9 matrix as you said. I made indexing
> mistake.
>

Great! Reducing the problem size always helps.

  Thanks,

    Matt


> Thank you for answering.
>
> Eda
>
> Matthew Knepley <knepley at gmail.com>, 24 Nis 2019 ?ar, 14:30 tarihinde
> ?unu yazd?:
>
>> On Wed, Apr 24, 2019 at 6:35 AM Stefano Zampini via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Dump the index sets and the matrix in binary and send them
>>>
>>
>> First, reduce your problem size to about 10.
>>
>>   Matt
>>
>>
>>> Il giorno mer 24 apr 2019 alle ore 13:21 Eda Oktay <
>>> eda.oktay at metu.edu.tr> ha scritto:
>>>
>>>> Since I am using square matrix, isn't the row local size of the matrix
>>>> equal to the local size of IS? They are both 2127 and 2126.
>>>>
>>>> Stefano Zampini <stefano.zampini at gmail.com>, 24 Nis 2019 ?ar, 13:15
>>>> tarihinde ?unu yazd?:
>>>>
>>>>> I don't understand your code. However, I suspect we have a bug here
>>>>> https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652
>>>>> The code assumes the number of leaves is A->rmap->n (the row local
>>>>> size of the matrix). Shouldn't this be the local size of the IS (rowp)?
>>>>> Barry?
>>>>>
>>>>> Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <
>>>>> eda.oktay at metu.edu.tr> ha scritto:
>>>>>
>>>>>> Dear Stefano,
>>>>>>
>>>>>> Thank you for answering. When I used MatSetSizes, I got an error
>>>>>> since the matrix is read from outside, but from the error I understood that
>>>>>> actually the local sizes are 2127 and 2126, so I misunderstood the problem.
>>>>>> I am sorry for my mistake.
>>>>>>
>>>>>> However, I still cannot understand where is the error. Because both
>>>>>> communicators and local sizes of IS and the matrix are the same. I still
>>>>>> get the same error in MatPermute:
>>>>>>
>>>>>> [0]PETSC ERROR: Argument out of range
>>>>>> [0]PETSC ERROR: Index -1081207334 is out of range
>>>>>> [0]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>> [0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL
>>>>>> on a arch-linux2-c-debug named 53d.wls.metu.edu.tr by edaoktay Wed
>>>>>> Apr 24 11:22:15 2019
>>>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>> --download-slepc --download-mpich
>>>>>> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in
>>>>>> /home/edaoktay/petsc-3.10.3/include/petscis.h
>>>>>> [0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> [0]PETSC ERROR: #3 MatPermute() line 4997 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>> [0]PETSC ERROR: #4 main() line 361 in
>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>
>>>>>> This is the part of my program:
>>>>>>
>>>>>>   ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);
>>>>>>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL);
>>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest
>>>>>> eigenvalue: %g\n",kr);CHKERRQ(ierr);
>>>>>>
>>>>>>   /* sort second smallest eigenvector */
>>>>>>
>>>>>>   ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);
>>>>>>   ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);
>>>>>>   for (i=0; i<siz;i++) idx[i] = i;
>>>>>>
>>>>>>   VecScatter ctx;
>>>>>>   Vec vout;
>>>>>>   VecScatterCreateToAll(vr,&ctx,&vout);
>>>>>>   VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>>>   VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);
>>>>>>   VecScatterDestroy(&ctx);
>>>>>>
>>>>>>   PetscScalar *avr;
>>>>>>   ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);
>>>>>>   ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr);
>>>>>>
>>>>>>   /*Select out a piece of the resulting indices idx on each process;
>>>>>> for example with two processes I think rank = 0 would get the first half of
>>>>>> the idx and rank = 1 would get the second half.*/
>>>>>>
>>>>>>   PetscMPIInt rank,size;
>>>>>>   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>>   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>>
>>>>>>   PetscInt mod;
>>>>>>   mod = siz % size;
>>>>>>
>>>>>>   PetscInt *idxx,ss;
>>>>>>   ss = (siz-mod)/size;
>>>>>>
>>>>>>   if (mod != 0){
>>>>>>     if (rank<mod){
>>>>>>         PetscMalloc1(ss+1,&idxx);
>>>>>>
>>>>>>     } else{
>>>>>>         PetscMalloc1(ss,&idxx);
>>>>>>     }
>>>>>>   } else{
>>>>>>       PetscMalloc1(ss,&idxx);
>>>>>>   }
>>>>>>
>>>>>>
>>>>>>     j =0;
>>>>>>     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>>       idxx[j] = idx[i];
>>>>>>      //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);
>>>>>>       j++;
>>>>>>
>>>>>>   if (mod != 0){
>>>>>>     if (rank<mod){
>>>>>>         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>>     }
>>>>>>   }
>>>>>>
>>>>>>   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>>
>>>>>>     if (mod != 0){
>>>>>>         if (rank<mod){
>>>>>>             ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>         } else{
>>>>>>             ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>         }
>>>>>>
>>>>>>     }else {
>>>>>>         ierr =
>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>     }
>>>>>>
>>>>>>   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>>
>>>>>> I printed IS and idxx, IS is a permutation and numbers seem to be
>>>>>> correct. The program also works when mod==0.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Eda
>>>>>>
>>>>>> Stefano Zampini <stefano.zampini at gmail.com>, 22 Nis 2019 Pzt, 18:13
>>>>>> tarihinde ?unu yazd?:
>>>>>>
>>>>>>> If you are using PETSC_DECIDE for the local sizes in MatSetSizes,
>>>>>>> the default local sizes should be 2127 and 2126.
>>>>>>>
>>>>>>> Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> ha scritto:
>>>>>>>
>>>>>>>> Thank you for your answers. I figured out that for a matrix of size
>>>>>>>> 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I
>>>>>>>> wrote the program such that the local sizes of IS 2127 and 2126.
>>>>>>>>
>>>>>>>> Is local size of Mat being 2127 on both processors correct? If it
>>>>>>>> is, then I will change the local size of IS but then it will exceed the
>>>>>>>> global size of Mat. Isn't this also a problem?
>>>>>>>>
>>>>>>>> Thanks a lot,
>>>>>>>>
>>>>>>>> Eda
>>>>>>>>
>>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>, 9 Nis 2019 Sal, 00:31
>>>>>>>> tarihinde ?unu yazd?:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>   Suggest printing out the IS with ISView after it is created and
>>>>>>>>> confirming that 1) it is a permutation and 2) that the size of the IS on
>>>>>>>>> each process matches the number of rows on that process.
>>>>>>>>>
>>>>>>>>>   Note from the manual page: The index sets should be on the same
>>>>>>>>> communicator as Mat and have the same local sizes.
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>> >
>>>>>>>>> > Hello again,
>>>>>>>>> >
>>>>>>>>> > I solved the problem for even numbered sized matrices. However
>>>>>>>>> when the matrix size is odd, then number of elements in each index set at
>>>>>>>>> each processor are different. (For example, for size 4253*4253 and 2
>>>>>>>>> processors, size of index set at processor 0 is 2127 where at processor 1,
>>>>>>>>> it is 2126) I think this is why, MatPermute again gives the same "Argument
>>>>>>>>> out of range" error. Index sets look like correct but I still did not get
>>>>>>>>> why I get this error.
>>>>>>>>> >
>>>>>>>>> > This is the part of my program:
>>>>>>>>> >
>>>>>>>>> >   PetscMPIInt rank,size;
>>>>>>>>> >   MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>>>>>>>> >   MPI_Comm_size(PETSC_COMM_WORLD, &size);
>>>>>>>>> >
>>>>>>>>> >   PetscInt mod;
>>>>>>>>> >   mod = siz % size;
>>>>>>>>> >
>>>>>>>>> >   PetscInt *idxx,ss;
>>>>>>>>> >   ss = (siz-mod)/size;
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> >   if (mod != 0){
>>>>>>>>> >     if (rank<mod){
>>>>>>>>> >         PetscMalloc1(ss+1,&idxx);
>>>>>>>>>
>>>>>>>>> >     } else{
>>>>>>>>> >         PetscMalloc1(ss,&idxx);
>>>>>>>>> >     }
>>>>>>>>> >   }
>>>>>>>>> >
>>>>>>>>> >   if (rank != size-1) {
>>>>>>>>> >     j =0;
>>>>>>>>> >     for (i=rank*ss; i<(rank+1)*ss; i++) {
>>>>>>>>> >       idxx[j] = idx[i];
>>>>>>>>> >       j++;
>>>>>>>>> >     }
>>>>>>>>> >
>>>>>>>>> >   } else {
>>>>>>>>> >
>>>>>>>>> >     j =0;
>>>>>>>>> >     for (i=rank*ss; i<siz; i++) {
>>>>>>>>> >       idxx[j] = idx[i];
>>>>>>>>> >       j++;
>>>>>>>>> >     }
>>>>>>>>> >
>>>>>>>>> >   }
>>>>>>>>> >
>>>>>>>>> >   if (mod != 0){
>>>>>>>>> >     if (rank<mod){
>>>>>>>>> >         idxx[ss+1] = idx[ss*size+rank+1];
>>>>>>>>> >     }
>>>>>>>>> >   }
>>>>>>>>> >
>>>>>>>>> >   /*Permute matrix L (spy(A(p1,p1))*/
>>>>>>>>> >
>>>>>>>>> >     if (mod != 0){
>>>>>>>>> >     if (rank<mod){
>>>>>>>>> >         ierr =
>>>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>>> >     } else{
>>>>>>>>> >         ierr =
>>>>>>>>> ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>>> >     }
>>>>>>>>> >   }
>>>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>>>> >
>>>>>>>>> >   ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr);
>>>>>>>>> >
>>>>>>>>> > And I get the following error even if I use MatSetOption :
>>>>>>>>> >
>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>>>> > [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc
>>>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>>>> to turn off this check
>>>>>>>>> > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>>>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>>>> arch-linux2-c-debug named dd2b.wls.metu.edu.tr by edaoktay Mon
>>>>>>>>> Apr  8 11:10:59 2019
>>>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>>>> --download-slepc --download-mpich
>>>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > [0]PETSC ERROR: #2 MatSetValues() line 1349 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>>> > [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > [0]PETSC ERROR: #4 MatPermute() line 4997 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>>> > [0]PETSC ERROR: #5 main() line 352 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>>>> > [0]PETSC ERROR: -f
>>>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>>>> > [0]PETSC ERROR: -unweighted
>>>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> >
>>>>>>>>> > Eda
>>>>>>>>> >
>>>>>>>>> > Eda Oktay <eda.oktay at metu.edu.tr>, 25 Mar 2019 Pzt, 13:53
>>>>>>>>> tarihinde ?unu yazd?:
>>>>>>>>> > I attached whole program I wrote where the problem is in line
>>>>>>>>> 285. One of the matrices I used was airfoil1_binary, included in the
>>>>>>>>> folder. Also, I included makefile. Is that what you want?
>>>>>>>>> >
>>>>>>>>> > Matthew Knepley <knepley at gmail.com>, 25 Mar 2019 Pzt, 13:41
>>>>>>>>> tarihinde ?unu yazd?:
>>>>>>>>> > That should not happen. Can you send in a small example that we
>>>>>>>>> can debug.
>>>>>>>>> >
>>>>>>>>> >   Thanks,
>>>>>>>>> >
>>>>>>>>> >     Matt
>>>>>>>>> >
>>>>>>>>> > On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>> > Hello,
>>>>>>>>> >
>>>>>>>>> > I am trying to permute a vector A using following lines:
>>>>>>>>> >
>>>>>>>>> > ierr =
>>>>>>>>> ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);
>>>>>>>>> >   ierr = ISSetPermutation(is);CHKERRQ(ierr);
>>>>>>>>> >   ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);
>>>>>>>>> >   ierr =
>>>>>>>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);
>>>>>>>>> >   ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr);
>>>>>>>>> >
>>>>>>>>> > However, in MatPermute line, I get the following error even if I
>>>>>>>>> used MatSetOption before this line:
>>>>>>>>> >
>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> > [0]PETSC ERROR: Argument out of range
>>>>>>>>> > [0]PETSC ERROR: New nonzero at (0,485) caused a malloc
>>>>>>>>> > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>>>>>>>>> to turn off this check
>>>>>>>>> > [0]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>>>>> > [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a
>>>>>>>>> arch-linux2-c-debug named 1232.wls.metu.edu.tr by edaoktay Mon
>>>>>>>>> Mar 25 12:15:14 2019
>>>>>>>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>>>>> --download-slepc --download-mpich
>>>>>>>>> > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>>> > [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>>>>>>>>> > [0]PETSC ERROR: #5 MatPermute() line 4997 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>>>>>>>>> > [0]PETSC ERROR: #6 main() line 285 in
>>>>>>>>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c
>>>>>>>>> > [0]PETSC ERROR: PETSc Option Table entries:
>>>>>>>>> > [0]PETSC ERROR: -f
>>>>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>>>>>>>>> > [0]PETSC ERROR: -mat_partitioning_type parmetis
>>>>>>>>> > [0]PETSC ERROR: -weighted
>>>>>>>>> > [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>>>> >
>>>>>>>>> > I'll be glad if you can help me.
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> >
>>>>>>>>> > Eda
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> > -- Norbert Wiener
>>>>>>>>> >
>>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Stefano
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Stefano
>>>>>
>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/077aecd6/attachment-0001.html>

From Tristan.Konolige at Colorado.EDU  Wed Apr 24 11:02:57 2019
From: Tristan.Konolige at Colorado.EDU (Tristan Konolige)
Date: Wed, 24 Apr 2019 10:02:57 -0600
Subject: [petsc-users] Passing a near nullspace through fieldsplit
Message-ID: <etPan.5cc088b2.4a2af24e.38e@colorado.edu>

Hello All,

I?m trying to pass a near nullspace through PCFieldSplit (schur) to GAMG (on the schur complement matrix). If I use MatSetNearNullSpace on my matrix, the near nullspace isn?t propagated through to GAMG. If I attach the near nullspace to the IS used to define the splits, it still doesn?t propagate through to GAMG. Am I missing the correct way to do this?

Thanks,
Tristan Konolige
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/77f21c64/attachment.html>

From bsmith at mcs.anl.gov  Wed Apr 24 11:31:26 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 24 Apr 2019 16:31:26 +0000
Subject: [petsc-users] Passing a near nullspace through fieldsplit
In-Reply-To: <etPan.5cc088b2.4a2af24e.38e@colorado.edu>
References: <etPan.5cc088b2.4a2af24e.38e@colorado.edu>
Message-ID: <47018258-BA7C-4E8E-9C77-2F84029839E3@anl.gov>



> On Apr 24, 2019, at 11:02 AM, Tristan Konolige via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello All,
> 
> I?m trying to pass a near nullspace through PCFieldSplit (schur) to GAMG (on the schur complement matrix). If I use MatSetNearNullSpace on my matrix, the near nullspace isn?t propagated through to GAMG. If I attach the near nullspace to the IS used to define the splits, it still doesn?t propagate through to GAMG. Am I missing the correct way to do this?

   You are probably not missing anything. 

The movement of null spaces and near null spaces may not be exactly as needed for your use case. The code is in
src/ksp/pc/impls/fieldsplit/fieldsplit.c start to search for NullSpace  (you only find a couple cases) 

  if (!jac->pmat) {
    Vec xtmp;

    ierr = MatCreateVecs(pc->pmat,&xtmp,NULL);CHKERRQ(ierr);
    ierr = PetscMalloc1(nsplit,&jac->pmat);CHKERRQ(ierr);
    ierr = PetscMalloc2(nsplit,&jac->x,nsplit,&jac->y);CHKERRQ(ierr);
    for (i=0; i<nsplit; i++) {
       ....
ierr = PetscObjectQuery((PetscObject) ilink->is, "nearnullspace", (PetscObject*) &sp);CHKERRQ(ierr);
      if (sp) {
        ierr = MatSetNearNullSpace(jac->pmat[i], sp);CHKERRQ(ierr);

    so we see it transferring the null space from each is to the pmat for each is. Later in the code

  for (i=0; i<nsplit; i++) {
    MatNullSpace sp;

    ierr = PetscObjectQuery((PetscObject) ilink->is, "nullspace", (PetscObject*) &sp);CHKERRQ(ierr);
    if (sp) {
      ierr = MatSetNullSpace(jac->mat[i], sp);CHKERRQ(ierr);
    }
    ilink = ilink->next;
  }

it seems to be similarly transferring the null space to each mat. Further in the code is

if (jac->schur) {
....
} else {
      const char   *Dprefix;
      char         schurprefix[256], schurmatprefix[256];
      char         schurtestoption[256];
      MatNullSpace sp;
....
 /* Note: this is not true in general */
      ierr = MatGetNullSpace(jac->mat[1], &sp);CHKERRQ(ierr);
      if (sp) {
        ierr = MatSetNullSpace(jac->schur, sp);CHKERRQ(ierr);
      }

  It seems to move the null space from mat[1] to jac->schur


  Now there may be other subroutines that get called that also transfer null spaces but I can't find them.

  You can try running in the debugger with a break point on MatGetNearNullSpace() MatSetNearNullSpace() to see where they are being passed around.

  I am not sure what matrix you are running GAMG on so it is unclear two me the path the near null space needs to take to get there. Is it jac->schur ?

   Good luck,

   Barry






> 
> Thanks,
> Tristan Konolige


From karl.linkui at gmail.com  Wed Apr 24 16:13:13 2019
From: karl.linkui at gmail.com (Karl Lin)
Date: Wed, 24 Apr 2019 16:13:13 -0500
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
Message-ID: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>

Hi, there

I have been trying to get a simple program run with the following code:

 12 int main(int argc,char **args)
 13 {
 14   PetscErrorCode ierr;
 15   Mat            A;
 16   Mat           AT;
 17   Mat            N;
 18   char           name[1024];
 19   char           vname[1024];
 20   char           pass[1024];
 21   PetscBool      flg;
 22   Vec            b,x,u,Ab,Au;
 23   PetscViewer    viewer;                        /* viewer */
 24   PetscMPIInt    rank,size;
 25
 26   KSP            QRsolver;
 27   PC             pc;
 28   PetscInt       its;
 29   PetscReal      norm;
 30
 31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
 32
 33   PetscInt       *cols, *dnz, *onz;
 34   PetscScalar    *vals;
 35
 36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
 37
 38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
 39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
 40
 41   PetscMalloc1(1, &dnz);
 42   PetscMalloc1(1, &onz);
 43
 44   dnz[0]=2;
 45   onz[0]=1;
 46
 47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz, &A);
CHKERRQ(ierr);
 48
 49   PetscMalloc1(2, &cols);
 50   PetscMalloc1(2, &vals);
 51
 52   jj = rank;
 53   cols[0]=0; cols[1]=1;
 54   vals[0]=1.0;vals[1]=1.0;
 55
 56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
 57
 58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
 59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
 60
 61   VecCreate(PETSC_COMM_WORLD,&x);
 62   VecSetSizes(x,PETSC_DECIDE,2);
 63   VecSetFromOptions(x);
 64
 65   VecCreate(PETSC_COMM_WORLD,&b);
 66   VecSetSizes(b,PETSC_DECIDE,1);
 67   VecSetFromOptions(b);
 68
 69   VecCreate(PETSC_COMM_WORLD,&u);
 70   VecSetSizes(u,PETSC_DECIDE,1);
 71   VecSetFromOptions(u);
 72
 73   VecSet(b, 2.0);
 74   VecSet(u, 0.0);
 75   VecSet(x, 0.0);
 76
 77   MatMult(A, x, u);
 78
 79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
 80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
 81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
 82
 83   VecAXPY(u,-1.0,b);

However, it always crashes at line 83 even with single process saying:
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019

I can't figure out why this would happen. The printout from VecView shows
every vec value is correct. I will greatly appreciate any tips.

Regards,
Karl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190424/4c344c8f/attachment.html>

From jczhang at mcs.anl.gov  Wed Apr 24 22:36:35 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 25 Apr 2019 03:36:35 +0000
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
In-Reply-To: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
References: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
Message-ID: <CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>

How many MPI ranks do you use? The following line is suspicious.  I guess you do not want a vector of global length 1.
66   VecSetSizes(b,PETSC_DECIDE,1);

--Junchao Zhang


On Wed, Apr 24, 2019 at 4:14 PM Karl Lin via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi, there

I have been trying to get a simple program run with the following code:

 12 int main(int argc,char **args)
 13 {
 14   PetscErrorCode ierr;
 15   Mat            A;
 16   Mat           AT;
 17   Mat            N;
 18   char           name[1024];
 19   char           vname[1024];
 20   char           pass[1024];
 21   PetscBool      flg;
 22   Vec            b,x,u,Ab,Au;
 23   PetscViewer    viewer;                        /* viewer */
 24   PetscMPIInt    rank,size;
 25
 26   KSP            QRsolver;
 27   PC             pc;
 28   PetscInt       its;
 29   PetscReal      norm;
 30
 31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
 32
 33   PetscInt       *cols, *dnz, *onz;
 34   PetscScalar    *vals;
 35
 36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
 37
 38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
 39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
 40
 41   PetscMalloc1(1, &dnz);
 42   PetscMalloc1(1, &onz);
 43
 44   dnz[0]=2;
 45   onz[0]=1;
 46
 47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz, &A); CHKERRQ(ierr);
 48
 49   PetscMalloc1(2, &cols);
 50   PetscMalloc1(2, &vals);
 51
 52   jj = rank;
 53   cols[0]=0; cols[1]=1;
 54   vals[0]=1.0;vals[1]=1.0;
 55
 56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
 57
 58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
 59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
 60
 61   VecCreate(PETSC_COMM_WORLD,&x);
 62   VecSetSizes(x,PETSC_DECIDE,2);
 63   VecSetFromOptions(x);
 64
 65   VecCreate(PETSC_COMM_WORLD,&b);
 66   VecSetSizes(b,PETSC_DECIDE,1);
 67   VecSetFromOptions(b);
 68
 69   VecCreate(PETSC_COMM_WORLD,&u);
 70   VecSetSizes(u,PETSC_DECIDE,1);
 71   VecSetFromOptions(u);
 72
 73   VecSet(b, 2.0);
 74   VecSet(u, 0.0);
 75   VecSet(x, 0.0);
 76
 77   MatMult(A, x, u);
 78
 79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
 80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
 81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
 82
 83   VecAXPY(u,-1.0,b);

However, it always crashes at line 83 even with single process saying:
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019

I can't figure out why this would happen. The printout from VecView shows every vec value is correct. I will greatly appreciate any tips.

Regards,
Karl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190425/54384d9d/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 25 00:34:55 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 25 Apr 2019 05:34:55 +0000
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
In-Reply-To: <CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>
References: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
	<CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>
Message-ID: <3D78BF69-8A20-483A-B4C9-0610FB5F8E57@anl.gov>


  I ran your problem fine on one process. no errors.

   Suggest you run under valgrind on your system: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind  or if that is not useful

   Run in the debugger with the option -start_in_debugger noxterm

  Good luck,

Barry



> On Apr 24, 2019, at 10:36 PM, Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> How many MPI ranks do you use? The following line is suspicious.  I guess you do not want a vector of global length 1. 
> 66   VecSetSizes(b,PETSC_DECIDE,1);
> 
> --Junchao Zhang
> 
> 
> On Wed, Apr 24, 2019 at 4:14 PM Karl Lin via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi, there
> 
> I have been trying to get a simple program run with the following code:
> 
>  12 int main(int argc,char **args)
>  13 {
>  14   PetscErrorCode ierr;
>  15   Mat            A;
>  16   Mat           AT;
>  17   Mat            N;
>  18   char           name[1024];
>  19   char           vname[1024];
>  20   char           pass[1024];
>  21   PetscBool      flg;
>  22   Vec            b,x,u,Ab,Au;
>  23   PetscViewer    viewer;                        /* viewer */
>  24   PetscMPIInt    rank,size;
>  25
>  26   KSP            QRsolver;
>  27   PC             pc;
>  28   PetscInt       its;
>  29   PetscReal      norm;
>  30
>  31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
>  32
>  33   PetscInt       *cols, *dnz, *onz;
>  34   PetscScalar    *vals;
>  35
>  36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
>  37
>  38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
>  39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
>  40
>  41   PetscMalloc1(1, &dnz);
>  42   PetscMalloc1(1, &onz);
>  43
>  44   dnz[0]=2;
>  45   onz[0]=1;
>  46
>  47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz, &A); CHKERRQ(ierr);
>  48
>  49   PetscMalloc1(2, &cols);
>  50   PetscMalloc1(2, &vals);
>  51
>  52   jj = rank;
>  53   cols[0]=0; cols[1]=1;
>  54   vals[0]=1.0;vals[1]=1.0;
>  55
>  56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
>  57
>  58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
>  59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
>  60
>  61   VecCreate(PETSC_COMM_WORLD,&x);
>  62   VecSetSizes(x,PETSC_DECIDE,2);
>  63   VecSetFromOptions(x);
>  64
>  65   VecCreate(PETSC_COMM_WORLD,&b);
>  66   VecSetSizes(b,PETSC_DECIDE,1);
>  67   VecSetFromOptions(b);
>  68
>  69   VecCreate(PETSC_COMM_WORLD,&u);
>  70   VecSetSizes(u,PETSC_DECIDE,1);
>  71   VecSetFromOptions(u);
>  72
>  73   VecSet(b, 2.0);
>  74   VecSet(u, 0.0);
>  75   VecSet(x, 0.0);
>  76
>  77   MatMult(A, x, u);
>  78
>  79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
>  80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
>  81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
>  82
>  83   VecAXPY(u,-1.0,b);
> 
> However, it always crashes at line 83 even with single process saying:
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019
> 
> I can't figure out why this would happen. The printout from VecView shows every vec value is correct. I will greatly appreciate any tips.
> 
> Regards,
> Karl
> 


From eda.oktay at metu.edu.tr  Thu Apr 25 06:16:13 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Thu, 25 Apr 2019 14:16:13 +0300
Subject: [petsc-users] SLEPc in PETSc 3.11.0
Message-ID: <CA+vu_XiF9cEB7wXk-3xSyjRjJncL+LN8XEyxXqrg=+4mBhpzQA@mail.gmail.com>

Hello everyone,

I am trying to run my program in petsc 3.11.0. It is currently working in
petsc 3.10.3.

I am using slepc in the program, so when I used 3.10.3 the slepc library is
in petsc-3.10.3/arch-linux2-c-debug. However, in 3.11.0, it is only in
petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc. I think that
is why, my program can not be compiled. I get the following error when I
type "make asil" to make the program asil.c:

/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/bin/mpicc -o asil.o -c
-Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-fstack-protector -fvisibility=hidden -g3
 -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include
-I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include
-I/home/edaoktay/petsc-3.11.0/include
-I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/include    `pwd`/asil.c
In file included from
/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcsys.h:45:0,
                 from
/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcst.h:16,
                 from
/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepceps.h:16,
                 from /home/edaoktay/petsc-3.11.0/src/eda/asil.c:10:
/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include/slepcconf.h:1:0:
error: unterminated #if
 #if !defined(__SLEPCCONF_H)

/home/edaoktay/petsc-3.11.0/lib/petsc/conf/rules:359: recipe for target
'asil.o' failed
make: *** [asil.o] Error 1


How can I get rid of this error? I have never get this when I used 3.10.3.

Thanks!

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190425/437b4132/attachment.html>

From jroman at dsic.upv.es  Thu Apr 25 06:18:20 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 25 Apr 2019 13:18:20 +0200
Subject: [petsc-users] SLEPc in PETSc 3.11.0
In-Reply-To: <CA+vu_XiF9cEB7wXk-3xSyjRjJncL+LN8XEyxXqrg=+4mBhpzQA@mail.gmail.com>
References: <CA+vu_XiF9cEB7wXk-3xSyjRjJncL+LN8XEyxXqrg=+4mBhpzQA@mail.gmail.com>
Message-ID: <889068B4-2D80-4E2E-8E88-86FB6C031639@dsic.upv.es>

Use SLEPc 3.11 with PETSc 3.11.


> El 25 abr 2019, a las 13:16, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hello everyone,
> 
> I am trying to run my program in petsc 3.11.0. It is currently working in petsc 3.10.3.
> 
> I am using slepc in the program, so when I used 3.10.3 the slepc library is in petsc-3.10.3/arch-linux2-c-debug. However, in 3.11.0, it is only in petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc. I think that is why, my program can not be compiled. I get the following error when I type "make asil" to make the program asil.c:
> 
> /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/bin/mpicc -o asil.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3   -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include  -I/home/edaoktay/petsc-3.11.0/include -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/include    `pwd`/asil.c
> In file included from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcsys.h:45:0,
>                  from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcst.h:16,
>                  from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepceps.h:16,
>                  from /home/edaoktay/petsc-3.11.0/src/eda/asil.c:10:
> /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include/slepcconf.h:1:0: error: unterminated #if
>  #if !defined(__SLEPCCONF_H)
>  
> /home/edaoktay/petsc-3.11.0/lib/petsc/conf/rules:359: recipe for target 'asil.o' failed
> make: *** [asil.o] Error 1
> 
> 
> How can I get rid of this error? I have never get this when I used 3.10.3.
> 
> Thanks!
> 
> Eda


From karl.linkui at gmail.com  Thu Apr 25 09:35:04 2019
From: karl.linkui at gmail.com (Karl Lin)
Date: Thu, 25 Apr 2019 09:35:04 -0500
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
In-Reply-To: <3D78BF69-8A20-483A-B4C9-0610FB5F8E57@anl.gov>
References: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
	<CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>
	<3D78BF69-8A20-483A-B4C9-0610FB5F8E57@anl.gov>
Message-ID: <CA+4MPWt2ziwmeoBywv0pNUOV+NMg_BHcH7_QgfhMwSS5sYqM8Q@mail.gmail.com>

Thanks for tip, I figured out the issue. I was linking to ilp64 instead of
lp64.

On Thu, Apr 25, 2019 at 12:34 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   I ran your problem fine on one process. no errors.
>
>    Suggest you run under valgrind on your system:
> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind  or if that
> is not useful
>
>    Run in the debugger with the option -start_in_debugger noxterm
>
>   Good luck,
>
> Barry
>
>
>
> > On Apr 24, 2019, at 10:36 PM, Zhang, Junchao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > How many MPI ranks do you use? The following line is suspicious.  I
> guess you do not want a vector of global length 1.
> > 66   VecSetSizes(b,PETSC_DECIDE,1);
> >
> > --Junchao Zhang
> >
> >
> > On Wed, Apr 24, 2019 at 4:14 PM Karl Lin via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi, there
> >
> > I have been trying to get a simple program run with the following code:
> >
> >  12 int main(int argc,char **args)
> >  13 {
> >  14   PetscErrorCode ierr;
> >  15   Mat            A;
> >  16   Mat           AT;
> >  17   Mat            N;
> >  18   char           name[1024];
> >  19   char           vname[1024];
> >  20   char           pass[1024];
> >  21   PetscBool      flg;
> >  22   Vec            b,x,u,Ab,Au;
> >  23   PetscViewer    viewer;                        /* viewer */
> >  24   PetscMPIInt    rank,size;
> >  25
> >  26   KSP            QRsolver;
> >  27   PC             pc;
> >  28   PetscInt       its;
> >  29   PetscReal      norm;
> >  30
> >  31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
> >  32
> >  33   PetscInt       *cols, *dnz, *onz;
> >  34   PetscScalar    *vals;
> >  35
> >  36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
> >  37
> >  38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
> >  39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
> >  40
> >  41   PetscMalloc1(1, &dnz);
> >  42   PetscMalloc1(1, &onz);
> >  43
> >  44   dnz[0]=2;
> >  45   onz[0]=1;
> >  46
> >  47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz,
> &A); CHKERRQ(ierr);
> >  48
> >  49   PetscMalloc1(2, &cols);
> >  50   PetscMalloc1(2, &vals);
> >  51
> >  52   jj = rank;
> >  53   cols[0]=0; cols[1]=1;
> >  54   vals[0]=1.0;vals[1]=1.0;
> >  55
> >  56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
> >  57
> >  58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
> >  59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
> >  60
> >  61   VecCreate(PETSC_COMM_WORLD,&x);
> >  62   VecSetSizes(x,PETSC_DECIDE,2);
> >  63   VecSetFromOptions(x);
> >  64
> >  65   VecCreate(PETSC_COMM_WORLD,&b);
> >  66   VecSetSizes(b,PETSC_DECIDE,1);
> >  67   VecSetFromOptions(b);
> >  68
> >  69   VecCreate(PETSC_COMM_WORLD,&u);
> >  70   VecSetSizes(u,PETSC_DECIDE,1);
> >  71   VecSetFromOptions(u);
> >  72
> >  73   VecSet(b, 2.0);
> >  74   VecSet(u, 0.0);
> >  75   VecSet(x, 0.0);
> >  76
> >  77   MatMult(A, x, u);
> >  78
> >  79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
> >  80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
> >  81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
> >  82
> >  83   VecAXPY(u,-1.0,b);
> >
> > However, it always crashes at line 83 even with single process saying:
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> > [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> OS X to find memory corruption errors
> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> > [0]PETSC ERROR: to get more information on the crash.
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Signal received
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019
> >
> > I can't figure out why this would happen. The printout from VecView
> shows every vec value is correct. I will greatly appreciate any tips.
> >
> > Regards,
> > Karl
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190425/70fa9f44/attachment.html>

From bsmith at mcs.anl.gov  Thu Apr 25 10:32:04 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 25 Apr 2019 15:32:04 +0000
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
In-Reply-To: <CA+4MPWt2ziwmeoBywv0pNUOV+NMg_BHcH7_QgfhMwSS5sYqM8Q@mail.gmail.com>
References: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
	<CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>
	<3D78BF69-8A20-483A-B4C9-0610FB5F8E57@anl.gov>
	<CA+4MPWt2ziwmeoBywv0pNUOV+NMg_BHcH7_QgfhMwSS5sYqM8Q@mail.gmail.com>
Message-ID: <1C81C63F-7AB5-44B3-A21C-65380F91DE40@mcs.anl.gov>


   Thanks for letting us know. The latest PETSc release automatically detects this issue at ./configure time. You might consider upgrading just to get the most stable software as we keep improving PETSc.

   Barry


> On Apr 25, 2019, at 9:35 AM, Karl Lin <karl.linkui at gmail.com> wrote:
> 
> Thanks for tip, I figured out the issue. I was linking to ilp64 instead of lp64.
> 
> On Thu, Apr 25, 2019 at 12:34 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   I ran your problem fine on one process. no errors.
> 
>    Suggest you run under valgrind on your system: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind  or if that is not useful
> 
>    Run in the debugger with the option -start_in_debugger noxterm
> 
>   Good luck,
> 
> Barry
> 
> 
> 
> > On Apr 24, 2019, at 10:36 PM, Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > How many MPI ranks do you use? The following line is suspicious.  I guess you do not want a vector of global length 1. 
> > 66   VecSetSizes(b,PETSC_DECIDE,1);
> > 
> > --Junchao Zhang
> > 
> > 
> > On Wed, Apr 24, 2019 at 4:14 PM Karl Lin via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > Hi, there
> > 
> > I have been trying to get a simple program run with the following code:
> > 
> >  12 int main(int argc,char **args)
> >  13 {
> >  14   PetscErrorCode ierr;
> >  15   Mat            A;
> >  16   Mat           AT;
> >  17   Mat            N;
> >  18   char           name[1024];
> >  19   char           vname[1024];
> >  20   char           pass[1024];
> >  21   PetscBool      flg;
> >  22   Vec            b,x,u,Ab,Au;
> >  23   PetscViewer    viewer;                        /* viewer */
> >  24   PetscMPIInt    rank,size;
> >  25
> >  26   KSP            QRsolver;
> >  27   PC             pc;
> >  28   PetscInt       its;
> >  29   PetscReal      norm;
> >  30
> >  31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
> >  32
> >  33   PetscInt       *cols, *dnz, *onz;
> >  34   PetscScalar    *vals;
> >  35
> >  36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
> >  37
> >  38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
> >  39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
> >  40
> >  41   PetscMalloc1(1, &dnz);
> >  42   PetscMalloc1(1, &onz);
> >  43
> >  44   dnz[0]=2;
> >  45   onz[0]=1;
> >  46
> >  47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz, &A); CHKERRQ(ierr);
> >  48
> >  49   PetscMalloc1(2, &cols);
> >  50   PetscMalloc1(2, &vals);
> >  51
> >  52   jj = rank;
> >  53   cols[0]=0; cols[1]=1;
> >  54   vals[0]=1.0;vals[1]=1.0;
> >  55
> >  56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
> >  57
> >  58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
> >  59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
> >  60
> >  61   VecCreate(PETSC_COMM_WORLD,&x);
> >  62   VecSetSizes(x,PETSC_DECIDE,2);
> >  63   VecSetFromOptions(x);
> >  64
> >  65   VecCreate(PETSC_COMM_WORLD,&b);
> >  66   VecSetSizes(b,PETSC_DECIDE,1);
> >  67   VecSetFromOptions(b);
> >  68
> >  69   VecCreate(PETSC_COMM_WORLD,&u);
> >  70   VecSetSizes(u,PETSC_DECIDE,1);
> >  71   VecSetFromOptions(u);
> >  72
> >  73   VecSet(b, 2.0);
> >  74   VecSet(u, 0.0);
> >  75   VecSet(x, 0.0);
> >  76
> >  77   MatMult(A, x, u);
> >  78
> >  79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
> >  80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
> >  81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
> >  82
> >  83   VecAXPY(u,-1.0,b);
> > 
> > However, it always crashes at line 83 even with single process saying:
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
> > [0]PETSC ERROR: to get more information on the crash.
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Signal received
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019
> > 
> > I can't figure out why this would happen. The printout from VecView shows every vec value is correct. I will greatly appreciate any tips.
> > 
> > Regards,
> > Karl
> > 
> 


From karl.linkui at gmail.com  Thu Apr 25 11:07:10 2019
From: karl.linkui at gmail.com (Karl Lin)
Date: Thu, 25 Apr 2019 11:07:10 -0500
Subject: [petsc-users] questions regarding simple petsc matrix vector
 operation
In-Reply-To: <1C81C63F-7AB5-44B3-A21C-65380F91DE40@mcs.anl.gov>
References: <CA+4MPWt0j_KeCD-+MDthBKdUtH0ESmQnOySOfyrvGc9uU9_RbQ@mail.gmail.com>
	<CA+MQGp_P7T4Ujjp2fbHc6DBj4YBfsHrTCE97UVA-hm5ddNCBnQ@mail.gmail.com>
	<3D78BF69-8A20-483A-B4C9-0610FB5F8E57@anl.gov>
	<CA+4MPWt2ziwmeoBywv0pNUOV+NMg_BHcH7_QgfhMwSS5sYqM8Q@mail.gmail.com>
	<1C81C63F-7AB5-44B3-A21C-65380F91DE40@mcs.anl.gov>
Message-ID: <CA+4MPWuiMn89uBKrBzh+zeKaQP7zfAbKw_ea4x5c8qKPt1in8g@mail.gmail.com>

Sure thank you.

On Thu, Apr 25, 2019 at 10:32 AM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    Thanks for letting us know. The latest PETSc release automatically
> detects this issue at ./configure time. You might consider upgrading just
> to get the most stable software as we keep improving PETSc.
>
>    Barry
>
>
> > On Apr 25, 2019, at 9:35 AM, Karl Lin <karl.linkui at gmail.com> wrote:
> >
> > Thanks for tip, I figured out the issue. I was linking to ilp64 instead
> of lp64.
> >
> > On Thu, Apr 25, 2019 at 12:34 AM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
> >
> >   I ran your problem fine on one process. no errors.
> >
> >    Suggest you run under valgrind on your system:
> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind  or if that
> is not useful
> >
> >    Run in the debugger with the option -start_in_debugger noxterm
> >
> >   Good luck,
> >
> > Barry
> >
> >
> >
> > > On Apr 24, 2019, at 10:36 PM, Zhang, Junchao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > >
> > > How many MPI ranks do you use? The following line is suspicious.  I
> guess you do not want a vector of global length 1.
> > > 66   VecSetSizes(b,PETSC_DECIDE,1);
> > >
> > > --Junchao Zhang
> > >
> > >
> > > On Wed, Apr 24, 2019 at 4:14 PM Karl Lin via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > > Hi, there
> > >
> > > I have been trying to get a simple program run with the following code:
> > >
> > >  12 int main(int argc,char **args)
> > >  13 {
> > >  14   PetscErrorCode ierr;
> > >  15   Mat            A;
> > >  16   Mat           AT;
> > >  17   Mat            N;
> > >  18   char           name[1024];
> > >  19   char           vname[1024];
> > >  20   char           pass[1024];
> > >  21   PetscBool      flg;
> > >  22   Vec            b,x,u,Ab,Au;
> > >  23   PetscViewer    viewer;                        /* viewer */
> > >  24   PetscMPIInt    rank,size;
> > >  25
> > >  26   KSP            QRsolver;
> > >  27   PC             pc;
> > >  28   PetscInt       its;
> > >  29   PetscReal      norm;
> > >  30
> > >  31   PetscInt       n1, n2, n3, np1, np2, np3, p, jj;
> > >  32
> > >  33   PetscInt       *cols, *dnz, *onz;
> > >  34   PetscScalar    *vals;
> > >  35
> > >  36   ierr = PetscInitialize(&argc,&args,0,help);if (ierr) return ierr;
> > >  37
> > >  38   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
> > >  39   ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
> > >  40
> > >  41   PetscMalloc1(1, &dnz);
> > >  42   PetscMalloc1(1, &onz);
> > >  43
> > >  44   dnz[0]=2;
> > >  45   onz[0]=1;
> > >  46
> > >  47   MatCreateMPIAIJMKL(PETSC_COMM_WORLD, 1, 2, 1, 2, 2, dnz, 2, onz,
> &A); CHKERRQ(ierr);
> > >  48
> > >  49   PetscMalloc1(2, &cols);
> > >  50   PetscMalloc1(2, &vals);
> > >  51
> > >  52   jj = rank;
> > >  53   cols[0]=0; cols[1]=1;
> > >  54   vals[0]=1.0;vals[1]=1.0;
> > >  55
> > >  56   MatSetValues(A, 1, &jj, 2, cols, vals, INSERT_VALUES);
> > >  57
> > >  58   MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
> > >  59   MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
> > >  60
> > >  61   VecCreate(PETSC_COMM_WORLD,&x);
> > >  62   VecSetSizes(x,PETSC_DECIDE,2);
> > >  63   VecSetFromOptions(x);
> > >  64
> > >  65   VecCreate(PETSC_COMM_WORLD,&b);
> > >  66   VecSetSizes(b,PETSC_DECIDE,1);
> > >  67   VecSetFromOptions(b);
> > >  68
> > >  69   VecCreate(PETSC_COMM_WORLD,&u);
> > >  70   VecSetSizes(u,PETSC_DECIDE,1);
> > >  71   VecSetFromOptions(u);
> > >  72
> > >  73   VecSet(b, 2.0);
> > >  74   VecSet(u, 0.0);
> > >  75   VecSet(x, 0.0);
> > >  76
> > >  77   MatMult(A, x, u);
> > >  78
> > >  79   VecView(x, PETSC_VIEWER_STDOUT_WORLD);
> > >  80   VecView(b, PETSC_VIEWER_STDOUT_WORLD);
> > >  81   VecView(u, PETSC_VIEWER_STDOUT_WORLD);
> > >  82
> > >  83   VecAXPY(u,-1.0,b);
> > >
> > > However, it always crashes at line 83 even with single process saying:
> > > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> > > [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > > [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> OS X to find memory corruption errors
> > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> > > [0]PETSC ERROR: to get more information on the crash.
> > > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > > [0]PETSC ERROR: Signal received
> > > [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019
> > >
> > > I can't figure out why this would happen. The printout from VecView
> shows every vec value is correct. I will greatly appreciate any tips.
> > >
> > > Regards,
> > > Karl
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190425/73ddea3c/attachment.html>

From hzhang at mcs.anl.gov  Thu Apr 25 14:47:30 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 25 Apr 2019 19:47:30 +0000
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBvNdLMfZhrXDBtdgzaikujQu98WarT6Qr9LGF5i0AAqUA@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<788f0293-4a5e-bae3-4a8d-10d92d0a16af@idris.fr>
	<CAJ98EDrfphzFNCd_Wb+ehxEvaE1OsrtbqhCG=z0C+kvNVFxvmA@mail.gmail.com>
	<d019386c-7a70-2b86-0d7a-c9880f2d3f43@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
	<CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
	<26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>
	<CAGCphBvNdLMfZhrXDBtdgzaikujQu98WarT6Qr9LGF5i0AAqUA@mail.gmail.com>
Message-ID: <CAGCphBs6Svo7vohR_x+MzHCwu_ycNW475s-F6P0PKH8xPzj0fg@mail.gmail.com>

Myriam:
Checking MatPtAP() in petsc-3.6.4, I realized that it uses different algorithm than petsc-10 and later versions. petsc-3.6 uses out-product for C=P^T * AP, while petsc-3.10 uses local transpose of P. petsc-3.10 accelerates data accessing, but doubles the memory of P.

Fande added two new implementations for MatPtAP() to petsc-master which use much smaller and scalable memories with slightly higher computing time (faster than hypre though). You may use these new implementations if you have concern on memory scalability. The option for these new implementation are:
-matptap_via allatonce
-matptap_via allatonce_merged

Hong

On Mon, Apr 15, 2019 at 12:10 PM hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov> <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Myriam:
Thank you very much for providing these results!
I have put effort to accelerate execution time and avoid using global sizes in PtAP, for which the algorithm of transpose of P_local and P_other likely doubles the memory usage. I'll try to investigate why it becomes unscalable.
Hong

Hi,

you'll find the new scaling attached (green line). I used the version 3.11 and the four scalability options :
-matptap_via scalable
-inner_diag_matmatmult_via scalable
-inner_offdiag_matmatmult_via scalable
-mat_freeintermediatedatastructures

The scaling is much better! The code even uses less memory for the smallest cases. There is still an increase for the larger one.

With regard to the time scaling, I used KSPView and LogView on the two previous scalings (blue and yellow lines) but not on the last one (green line). So we can't really compare them, am I right? However, we can see that the new time scaling looks quite good. It slightly increases from ~8s to ~27s.

Unfortunately, the computations are expensive so I would like to avoid re-run them if possible. How relevant would be a proper time scaling for you?

Myriam

Le 04/12/19 ? 18:18, Zhang, Hong a ?crit :
Myriam :
Thanks for your effort. It will help us improve PETSc.
Hong

Hi all,

I used the wrong script, that's why it diverged... Sorry about that.
I tried again with the right script applied on a tiny problem (~200
elements). I can see a small difference in memory usage (gain ~ 1mB).
when adding the -mat_freeintermediatestructures option. I still have to
execute larger cases to plot the scaling. The supercomputer I am used to
run my jobs on is really busy at the moment so it takes a while. I hope
I'll send you the results on Monday.

Thanks everyone,

Myriam


Le 04/11/19 ? 06:01, Jed Brown a ?crit :
> "Zhang, Hong" <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> writes:
>
>> Jed:
>>>> Myriam,
>>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.
>>> And yet if turning it on causes divergence, that would imply a bug.
>>> Hong, are you able to reproduce the experiment to see the memory
>>> scaling?
>> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.
> I have now compiled and run her code locally.
>
> Myriam, thanks for your last mail adding configuration and removing the
> MemManager.h dependency.  I ran with and without
> -mat_freeintermediatedatastructures and don't see a difference in
> convergence.  What commands did you run to observe that difference?

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--




--
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190425/4c0c5fab/attachment.html>

From yjwu16 at gmail.com  Fri Apr 26 07:24:31 2019
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Fri, 26 Apr 2019 20:24:31 +0800
Subject: [petsc-users] Problems about fd_coloring
Message-ID: <CAAMEhBgp=qGQh-TSaSsPtBbopjix3EaQQxCo=qF4e5gUu5yi+A@mail.gmail.com>

Dear PETSc developers:
Hi,
I want to try some of the operations of matrix coloring. I have written a
model of a system of non-linear equations and solved it using - snes_fd, so
there is no Compute_Jacobian routine. Previous developers suggested that I
use:
 -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greendy
to improve computational efficiency. I know little about coloring. It seems
to be a way to store the matrix structure, so that when the finite
difference routine is used, the difference can be calculated only at the
position of non-zero elements of Jacobian matrix, avoiding the difference
calculation of "0" elements. I don't know if I understand this correctly.


My program is a two-dimensional problem, but it does not use the DM object
to allocate the solution vector X. Is there any problem with fd_coloring?


For users like me who don't know coloring, how should I get some usage
methods? It seems that the instructions on manual are somewhat brief.

In addition, because Matrix Free needs an efficient Preconditioner and
because some complex parameter variables make it impossible for me to
analytically construct Jacobian matrices, the finite difference
approximation of Jacobian matrices may be the most appropriate method at
present.The reason I want to know about fd_coloring is to improve
computational efficiency.

I hope to get your advice.

Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190426/a0fdc39e/attachment.html>

From mfadams at lbl.gov  Fri Apr 26 07:45:19 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 26 Apr 2019 08:45:19 -0400
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4GkPNv4k501_+CpB2yXtxV812PgJLxPLKkvM3zx=3_mLjg@mail.gmail.com>
	<fb8151ff-aadd-bb5e-1f1d-b40a5565f90b@idris.fr>
	<CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
Message-ID: <CADOhEh65RBKb7spMo02BO_SWKzxaueM4kq_sNPD1cQz0EpTVmA@mail.gmail.com>

> Mark, what is the option she can give to output all the GAMG data?
>
>

-info and then grep on GAMG.

This will print the number of non-zeros per row, which is useful. The
memory size of the matrices will also give you data on this.


> Also, run using -ksp_view. GAMG will report all the sizes of its grids, so
> it should be easy to see
> if the coarse grid sizes are increasing, and also what the effect of the
> threshold value is.
>
>   Thanks,
>
>      Matt
>
>> Thanks
>> Myriam
>>
>> Le 03/02/19 ? 02:27, Matthew Knepley a ?crit :
>>
>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
>>> to 3.10, this code has a bad memory scaling.
>>>
>>> To report this issue, I took the PETSc script ex42.c and slightly
>>> modified it so that the KSP and PC configurations are the same as in my
>>> code. In particular, I use a "personnalised" multi-grid method. The
>>> modifications are indicated by the keyword "TopBridge" in the attached
>>> scripts.
>>>
>>> To plot the memory (weak) scaling, I ran four calculations for each
>>> script with increasing problem sizes and computations cores:
>>>
>>> 1. 100,000 elts on 4 cores
>>> 2. 1 million elts on 40 cores
>>> 3. 10 millions elts on 400 cores
>>> 4. 100 millions elts on 4,000 cores
>>>
>>> The resulting graph is also attached. The scaling using PETSc 3.10
>>> clearly deteriorates for large cases, while the one using PETSc 3.6 is
>>> robust.
>>>
>>> After a few tests, I found that the scaling is mostly sensitive to the
>>> use of the AMG method for the coarse grid (line 1780 in
>>> main_ex42_petsc36.cc). In particular, the performance strongly
>>> deteriorates when commenting lines 1777 to 1790 (in
>>> main_ex42_petsc36.cc).
>>>
>>> Do you have any idea of what changed between version 3.6 and version
>>> 3.10 that may imply such degradation?
>>>
>>
>> I believe the default values for PCGAMG changed between versions. It
>> sounds like the coarsening rate
>> is not great enough, so that these grids are too large. This can be set
>> using:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>
>> There is some explanation of this effect on that page. Let us know if
>> setting this does not correct the situation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Let me know if you need further information.
>>>
>>> Best,
>>>
>>> Myriam Peyrounette
>>>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190426/73d64b39/attachment-0001.html>

From mfadams at lbl.gov  Fri Apr 26 07:47:00 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 26 Apr 2019 08:47:00 -0400
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <3daa411c-d2c4-53d3-ff7e-c14429b40e49@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4GkPNv4k501_+CpB2yXtxV812PgJLxPLKkvM3zx=3_mLjg@mail.gmail.com>
	<fb8151ff-aadd-bb5e-1f1d-b40a5565f90b@idris.fr>
	<CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
	<3daa411c-d2c4-53d3-ff7e-c14429b40e49@idris.fr>
Message-ID: <CADOhEh4O7FbTiu08_=5XY=Ygs2D8A9qhv_bd_=j+_po=M=1jPw@mail.gmail.com>

Increasing the threshold should increase the size of the coarse grids, but
yours are decreasing. I'm puzzled by that.

On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette <
myriam.peyrounette at idris.fr> wrote:

> I used PCView to display the size of the linear system in each level of
> the MG. You'll find the outputs attached to this mail (zip file) for both
> the default threshold value and a value of 0.1, and for both 3.6 and 3.10
> PETSc versions.
>
> For convenience, I summarized the information in a graph, also attached
> (png file).
>
> As you can see, there are slight differences between the two versions but
> none is critical, in my opinion. Do you see anything suspicious in the
> outputs?
>
> + I can't find the default threshold value. Do you know where I can find
> it?
>
> Thanks for the follow-up
>
> Myriam
>
> Le 03/05/19 ? 14:06, Matthew Knepley a ?crit :
>
> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette <
> myriam.peyrounette at idris.fr> wrote:
>
>> Hi Matt,
>>
>> I plotted the memory scalings using different threshold values. The two
>> scalings are slightly translated (from -22 to -88 mB) but this gain is
>> neglectable. The 3.6-scaling keeps being robust while the 3.10-scaling
>> deteriorates.
>>
>> Do you have any other suggestion?
>>
> Mark, what is the option she can give to output all the GAMG data?
>
> Also, run using -ksp_view. GAMG will report all the sizes of its grids, so
> it should be easy to see
> if the coarse grid sizes are increasing, and also what the effect of the
> threshold value is.
>
>   Thanks,
>
>      Matt
>
>> Thanks
>> Myriam
>>
>> Le 03/02/19 ? 02:27, Matthew Knepley a ?crit :
>>
>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
>>> to 3.10, this code has a bad memory scaling.
>>>
>>> To report this issue, I took the PETSc script ex42.c and slightly
>>> modified it so that the KSP and PC configurations are the same as in my
>>> code. In particular, I use a "personnalised" multi-grid method. The
>>> modifications are indicated by the keyword "TopBridge" in the attached
>>> scripts.
>>>
>>> To plot the memory (weak) scaling, I ran four calculations for each
>>> script with increasing problem sizes and computations cores:
>>>
>>> 1. 100,000 elts on 4 cores
>>> 2. 1 million elts on 40 cores
>>> 3. 10 millions elts on 400 cores
>>> 4. 100 millions elts on 4,000 cores
>>>
>>> The resulting graph is also attached. The scaling using PETSc 3.10
>>> clearly deteriorates for large cases, while the one using PETSc 3.6 is
>>> robust.
>>>
>>> After a few tests, I found that the scaling is mostly sensitive to the
>>> use of the AMG method for the coarse grid (line 1780 in
>>> main_ex42_petsc36.cc). In particular, the performance strongly
>>> deteriorates when commenting lines 1777 to 1790 (in
>>> main_ex42_petsc36.cc).
>>>
>>> Do you have any idea of what changed between version 3.6 and version
>>> 3.10 that may imply such degradation?
>>>
>>
>> I believe the default values for PCGAMG changed between versions. It
>> sounds like the coarsening rate
>> is not great enough, so that these grids are too large. This can be set
>> using:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>
>> There is some explanation of this effect on that page. Let us know if
>> setting this does not correct the situation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Let me know if you need further information.
>>>
>>> Best,
>>>
>>> Myriam Peyrounette
>>>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>
>
>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190426/94d9e4f2/attachment.html>

From mfadams at lbl.gov  Fri Apr 26 07:58:59 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 26 Apr 2019 08:58:59 -0400
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <d7928f62-f6fb-5606-8285-ec935b302cc9@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4GkPNv4k501_+CpB2yXtxV812PgJLxPLKkvM3zx=3_mLjg@mail.gmail.com>
	<fb8151ff-aadd-bb5e-1f1d-b40a5565f90b@idris.fr>
	<CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
	<3daa411c-d2c4-53d3-ff7e-c14429b40e49@idris.fr>
	<CAMYG4G=-e6rG+4w+WOx+iFU_GafBac9Y54ZpCZ+7HTnmNJrZiA@mail.gmail.com>
	<C2DF124D-E9CC-4847-84A7-D07F5864B8AE@anl.gov>
	<87sgw0pqlj.fsf@jedbrown.org>
	<b87e33d6-5b3e-0a19-6d03-a089bba9d57e@idris.fr>
	<87ftrx9uga.fsf@jedbrown.org>
	<CADOhEh4FqanoNjfy1rKgLp3LShEiDg+79E-Y_3jTdgFrFrcuRg@mail.gmail.com>
	<d7928f62-f6fb-5606-8285-ec935b302cc9@idris.fr>
Message-ID: <CADOhEh4YgkrY-CQZ8YdM5uY-y-E=zSqGs0d8P-V2H5KtTeotxw@mail.gmail.com>

On Mon, Mar 11, 2019 at 6:32 AM Myriam Peyrounette <
myriam.peyrounette at idris.fr> wrote:

> Hi,
>
> good point, I changed the 3.10 version so that it is configured with
> --with-debugging=0. You'll find attached the output of the new LogView. The
> execution time is reduced (although still not as good as 3.6) but I can't
> see any improvement with regard to memory.
>
> You'll also find attached the grep GAMG on -info outputs for both
> versions. There are slight differences in grid dimensions or nnz values,
> but is it significant?
>

No. And the GAMG runs seem to be on a 13x13x13 grid.

The nnz goes down a lot here, which is odd, but the problem is so small
that we are probably just seeing boundary effects.


> Thanks,
>
> Myriam
>
>
>
> Le 03/08/19 ? 23:23, Mark Adams a ?crit :
>
> Just seeing this now. It is hard to imagine how bad GAMG could be on a
> coarse grid, but you can run with -info and grep on GAMG and send that. You
> will see listing of levels, number of equations and number of non-zeros
> (nnz). You can send that and I can get some sense of GAMG is going nuts.
>
> Mark
>
> On Fri, Mar 8, 2019 at 11:56 AM Jed Brown via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> It may not address the memory issue, but can you build 3.10 with the
>> same options you used for 3.6?  It is currently a debugging build:
>>
>>               ##########################################################
>>               #                                                        #
>>               #                       WARNING!!!                       #
>>               #                                                        #
>>               #   This code was compiled with a debugging option.      #
>>               #   To get timing results run ./configure                #
>>               #   using --with-debugging=no, the performance will      #
>>               #   be generally two or three times faster.              #
>>               #                                                        #
>>               ##########################################################
>>
>>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190426/6af53199/attachment.html>

From bsmith at mcs.anl.gov  Fri Apr 26 10:02:18 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 26 Apr 2019 15:02:18 +0000
Subject: [petsc-users] Problems about fd_coloring
In-Reply-To: <CAAMEhBgp=qGQh-TSaSsPtBbopjix3EaQQxCo=qF4e5gUu5yi+A@mail.gmail.com>
References: <CAAMEhBgp=qGQh-TSaSsPtBbopjix3EaQQxCo=qF4e5gUu5yi+A@mail.gmail.com>
Message-ID: <5FC35353-B739-41D5-A6D1-E4AE0ADFF708@anl.gov>



> On Apr 26, 2019, at 7:24 AM, Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear PETSc developers:            
> Hi,            
> I want to try some of the operations of matrix coloring. I have written a model of a system of non-linear equations and solved it using - snes_fd, so there is no Compute_Jacobian routine. Previous developers suggested that I use:
>  -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greendy 
> to improve computational efficiency. I know little about coloring. It seems to be a way to store the matrix structure, so that when the finite difference routine is used, the difference can be calculated only at the position of non-zero elements of Jacobian matrix, avoiding the difference calculation of "0" elements. I don't know if I understand this correctly. 

   This is close enough understanding.
>           
> 
> My program is a two-dimensional problem, but it does not use the DM object to allocate the solution vector X. Is there any problem with fd_coloring?   

  In order to use coloring you MUST provide the Jacobian matrix with its correct nonzero structure (you don't need to provide the numerical values). The fd coloring code then uses the nonzero structure and the SNES function you provide to compute the Jacobian efficiently. 

  Depending on your discretization it is either trivial or somewhat straight forward to provide the Jacobian nonzero structure. For example for finite differences on a 2d grid you can figure out which entires in the matrix are nonzero based on the finite difference stencil. For finite elements it is slightly more involved but essentially for each row of the matrix there are nonzero entries for all columns that that are connected by the finite element discretization (that is if you were to compute the Jacobian explicitly and get a nonzero value then you get one a nonzero location in the matrix). 

  Now in parallel one must be careful and insure you handle the nonzero structure correctly for vertices that are not on the same processor.

  Barry

>         
> 
> For users like me who don't know coloring, how should I get some usage methods? It seems that the instructions on manual are somewhat brief.
> 
> In addition, because Matrix Free needs an efficient Preconditioner and because some complex parameter variables make it impossible for me to analytically construct Jacobian matrices, the finite difference approximation of Jacobian matrices may be the most appropriate method at present.The reason I want to know about fd_coloring is to improve computational efficiency.
> 
> I hope to get your advice.
> 
> Thanks,
> Yingjie


From zonexo at gmail.com  Sat Apr 27 10:39:49 2019
From: zonexo at gmail.com (TAY wee-beng)
Date: Sat, 27 Apr 2019 23:39:49 +0800
Subject: [petsc-users] DMDACreate3d for 2 different sizes
Message-ID: <c94f5cd8-26c8-657f-72ef-d65e1477d940@gmail.com>

Hi,

I need to use DMDACreate3d to create objects da and da_psr which have 2 
different sizes:

call 
DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,&

size_z,one,PETSC_DECIDE,PETSC_DECIDE,one,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_p,ierr)


call 
DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x_psr,size_y_psr,&

size_z_psr,one,PETSC_DECIDE,PETSC_DECIDE,one,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_p_psr,ierr)


size_x/y/z_psr is actually half of size_x/y/z

However, running the 2nd line gave error:

[0]PETSC ERROR: Arguments are incompatible
[0]PETSC ERROR: Ownership ranges sum to 76 but global dimension is 38
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trou
ble shooting.

size_x is 76 but size_x_prs is 38.

So do I need to create 2 comm (MPI communicator) if I need to create 
da_p_psr? are using different values of size_x_psr,size_y_psr, 
size_z_psr not enough?

-- 

Thank you very much.

Yours sincerely,

================================================
TAY Wee-Beng ??? (Zheng Weiming)
Personal research webpage: _http://tayweebeng.wixsite.com/website_
Youtube research showcase: _https://goo.gl/PtvdwQ_
linkedin: _https://www.linkedin.com/in/tay-weebeng_
================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190427/6bb13468/attachment.html>

From knepley at gmail.com  Sat Apr 27 10:49:52 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 27 Apr 2019 11:49:52 -0400
Subject: [petsc-users] DMDACreate3d for 2 different sizes
In-Reply-To: <c94f5cd8-26c8-657f-72ef-d65e1477d940@gmail.com>
References: <c94f5cd8-26c8-657f-72ef-d65e1477d940@gmail.com>
Message-ID: <CAMYG4Gmkt5s76-uqyQUcbbn3Ju2Nt-uP-EhKxHcit_yFkVpMCw@mail.gmail.com>

On Sat, Apr 27, 2019 at 11:41 AM TAY wee-beng via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I need to use DMDACreate3d to create objects da and da_psr which have 2
> different sizes:
>
> call
> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,&
>
>
> size_z,one,PETSC_DECIDE,PETSC_DECIDE,one,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_p,ierr)
>
>
> call
> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x_psr,size_y_psr,&
>
>
> size_z_psr,one,PETSC_DECIDE,PETSC_DECIDE,one,stencil_width,lx,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da_p_psr,ierr)
>
You left in 'ix' for the second call, but it cannot be right.

  Matt

> size_x/y/z_psr is actually half of size_x/y/z
>
> However, running the 2nd line gave error:
>
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: Ownership ranges sum to 76 but global dimension is 38
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trou
> ble shooting.
>
> size_x is 76 but size_x_prs is 38.
>
> So do I need to create 2 comm (MPI communicator) if I need to create
> da_p_psr? are using different values of size_x_psr,size_y_psr, size_z_psr
> not enough?
> --
>
> Thank you very much.
>
> Yours sincerely,
>
> ================================================
> TAY Wee-Beng ??? (Zheng Weiming)
> Personal research webpage: *http://tayweebeng.wixsite.com/website
> <http://tayweebeng.wixsite.com/website>*
> Youtube research showcase: *https://goo.gl/PtvdwQ <https://goo.gl/PtvdwQ>*
> linkedin: *https://www.linkedin.com/in/tay-weebeng
> <https://www.linkedin.com/in/tay-weebeng>*
> ================================================
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190427/9a76fb01/attachment.html>

From bsmith at mcs.anl.gov  Sat Apr 27 21:19:10 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 28 Apr 2019 02:19:10 +0000
Subject: [petsc-users] SLEPc in PETSc 3.11.0
In-Reply-To: <889068B4-2D80-4E2E-8E88-86FB6C031639@dsic.upv.es>
References: <CA+vu_XiF9cEB7wXk-3xSyjRjJncL+LN8XEyxXqrg=+4mBhpzQA@mail.gmail.com>
	<889068B4-2D80-4E2E-8E88-86FB6C031639@dsic.upv.es>
Message-ID: <849D8BFD-41A6-48D4-9E10-62ABA54C7968@anl.gov>


  Delete the current PETSc directory and upgrade to PETSc 3.11.1 or better yet download it via git: https://www.mcs.anl.gov/petsc/download/index.html
and it should automatically use the right version of SLEPc when you use --download-slepc 

  If that fails send configure.log and make.log to petsc-maint at mcs.anl.gov

   Barry


> On Apr 25, 2019, at 6:18 AM, Jose E. Roman via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Use SLEPc 3.11 with PETSc 3.11.
> 
> 
>> El 25 abr 2019, a las 13:16, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
>> 
>> Hello everyone,
>> 
>> I am trying to run my program in petsc 3.11.0. It is currently working in petsc 3.10.3.
>> 
>> I am using slepc in the program, so when I used 3.10.3 the slepc library is in petsc-3.10.3/arch-linux2-c-debug. However, in 3.11.0, it is only in petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc. I think that is why, my program can not be compiled. I get the following error when I type "make asil" to make the program asil.c:
>> 
>> /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/bin/mpicc -o asil.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3   -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include  -I/home/edaoktay/petsc-3.11.0/include -I/home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/include    `pwd`/asil.c
>> In file included from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcsys.h:45:0,
>>                 from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepcst.h:16,
>>                 from /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/include/slepceps.h:16,
>>                 from /home/edaoktay/petsc-3.11.0/src/eda/asil.c:10:
>> /home/edaoktay/petsc-3.11.0/arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/include/slepcconf.h:1:0: error: unterminated #if
>> #if !defined(__SLEPCCONF_H)
>> 
>> /home/edaoktay/petsc-3.11.0/lib/petsc/conf/rules:359: recipe for target 'asil.o' failed
>> make: *** [asil.o] Error 1
>> 
>> 
>> How can I get rid of this error? I have never get this when I used 3.10.3.
>> 
>> Thanks!
>> 
>> Eda
> 


From boyceg at email.unc.edu  Mon Apr 29 19:10:32 2019
From: boyceg at email.unc.edu (Griffith, Boyce Eugene)
Date: Tue, 30 Apr 2019 00:10:32 +0000
Subject: [petsc-users] ASM vs GASM
Message-ID: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>

I am trying to setup a GASM preconditioner and am running into some difficulty getting convergence even with exact subdomain solvers. I am just doing things in serial now, and tried switching to ASM, and get convergence. I am wondering if I am misunderstanding the GASM interface. If inner_ises and outer_ises define the ?inner? and ?outer? subdomains for GASM, should GASM and ASM do the same things when configured via:

	PCGASMSetSubdomains(pc, n_subdomains, inner_ises, outer_ises);

and

	PCASMSetLocalSubdomains(pc, n_subdomains, outer_ises, inner_ises);

Thanks!

? Boyce

From bsmith at mcs.anl.gov  Mon Apr 29 19:39:51 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 30 Apr 2019 00:39:51 +0000
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
Message-ID: <AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>


  I believe so. Probably you've already done this but you can look at src/ksp/ksp/examples/tutorials ex62.c and ex64.c for sample usage.

  Do you get the same (correct) behavior for both when you use inner_ises but no outer (let GASM/ASM determine it)? What if you use the same outer as inner?

  Note that for ASM the "inner" subdomains can overlap while for GASM they cannot overlap (and of course their union needs to be all degrees of freedom).

  GASM is usually used where each inner is exactly several processes but it should work more generally as well.

   Barry


> On Apr 29, 2019, at 7:10 PM, Griffith, Boyce Eugene via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> I am trying to setup a GASM preconditioner and am running into some difficulty getting convergence even with exact subdomain solvers. I am just doing things in serial now, and tried switching to ASM, and get convergence. I am wondering if I am misunderstanding the GASM interface. If inner_ises and outer_ises define the ?inner? and ?outer? subdomains for GASM, should GASM and ASM do the same things when configured via:
> 
> 	PCGASMSetSubdomains(pc, n_subdomains, inner_ises, outer_ises);
> 
> and
> 
> 	PCASMSetLocalSubdomains(pc, n_subdomains, outer_ises, inner_ises);
> 
> Thanks!
> 
> ? Boyce


From boyceg at gmail.com  Mon Apr 29 20:47:34 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Mon, 29 Apr 2019 21:47:34 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
Message-ID: <A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>



> On Apr 29, 2019, at 8:39 PM, Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
>  I believe so. Probably you've already done this but you can look at src/ksp/ksp/examples/tutorials ex62.c and ex64.c for sample usage.

Yep!

>  Do you get the same (correct) behavior for both when you use inner_ises but no outer (let GASM/ASM determine it)? What if you use the same outer as inner?

I think that I had a bug in setting up the overlapping subdomains. Now GASM seems to work, but the convergence history does not seem to depend on how I set the outer IS, which seems surprising. In particular, with GASM, I?ve tried setting the outer IS to include overlaps; setting the outer IS to be the inner IS; and setting the outer IS to NULL ? all converge in 2 iterations using exact subdomain solves. (This is a problem with only 2 subdomains.)

ASM gives different results. Specifically, if I let ASM determine the overlaps automatically, then it converges in 4 iterations. If I use the inner IS and the overlapping IS, then it takes nearly 40.

>  Note that for ASM the "inner" subdomains can overlap while for GASM they cannot overlap (and of course their union needs to be all degrees of freedom).

I think that I have been setting up the inner subdomains correctly for GASM. I have some doubts about how I am setting up the overlapping regions, and so I think I will just let GASM setup the overlaps for now.

>  GASM is usually used where each inner is exactly several processes but it should work more generally as well.

This is a situation where I have different PDEs on two subdomains with a flux continuity condition along the interface. One subdomain has a scalar PDE (just Laplace?s equation), the other has a vector PDE. I am trying to use GASM to split up the problems, and then to use GAMG on the scalar domain and FIELDSPLIT on the vector domain. I already know that FIELDSPLIT can be made to work well on the vector PDE.

Thanks!

? Boyce

>   Barry
> 
> 
>> On Apr 29, 2019, at 7:10 PM, Griffith, Boyce Eugene via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> I am trying to setup a GASM preconditioner and am running into some difficulty getting convergence even with exact subdomain solvers. I am just doing things in serial now, and tried switching to ASM, and get convergence. I am wondering if I am misunderstanding the GASM interface. If inner_ises and outer_ises define the ?inner? and ?outer? subdomains for GASM, should GASM and ASM do the same things when configured via:
>> 
>> 	PCGASMSetSubdomains(pc, n_subdomains, inner_ises, outer_ises);
>> 
>> and
>> 
>> 	PCASMSetLocalSubdomains(pc, n_subdomains, outer_ises, inner_ises);
>> 
>> Thanks!
>> 
>> ? Boyce
> 


From myriam.peyrounette at idris.fr  Tue Apr 30 03:25:41 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Tue, 30 Apr 2019 10:25:41 +0200
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAGCphBs6Svo7vohR_x+MzHCwu_ycNW475s-F6P0PKH8xPzj0fg@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
	<CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
	<26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>
	<CAGCphBvNdLMfZhrXDBtdgzaikujQu98WarT6Qr9LGF5i0AAqUA@mail.gmail.com>
	<CAGCphBs6Svo7vohR_x+MzHCwu_ycNW475s-F6P0PKH8xPzj0fg@mail.gmail.com>
Message-ID: <31946231-4948-fc6a-093a-7ed8f00f3579@idris.fr>

Hi,

that's really good news for us, thanks! I will plot again the memory
scaling using these new options and let you know. Next week I hope.

Before that, I just need to clarify the situation. Throughout our
discussions, we mentionned a number of options concerning the scalability:

-matptatp_via scalable
-inner_diag_matmatmult_via scalable
-inner_diag_matmatmult_via scalable
-mat_freeintermediatedatastructures
-matptap_via allatonce
-matptap_via allatonce_merged

Which ones of them are compatible? Should I use all of them at the same
time? Is there redundancy?

Thanks,

Myriam


Le 04/25/19 ? 21:47, Zhang, Hong a ?crit?:
> Myriam:
> Checking MatPtAP() in petsc-3.6.4, I realized that it uses different
> algorithm than petsc-10 and later versions. petsc-3.6 uses out-product
> for C=P^T * AP, while petsc-3.10 uses local transpose of P. petsc-3.10
> accelerates data accessing, but doubles the memory of P.?
>
> Fande added two new implementations for MatPtAP() to petsc-master
> which use much smaller and scalable memories with slightly higher
> computing time (faster than hypre though). You may use these new
> implementations if you have concern on memory scalability. The option
> for these new implementation are:?
> -matptap_via allatonce
> -matptap_via allatonce_merged
>
> Hong
>
> On Mon, Apr 15, 2019 at 12:10 PM hzhang at mcs.anl.gov
> <mailto:hzhang at mcs.anl.gov> <hzhang at mcs.anl.gov
> <mailto:hzhang at mcs.anl.gov>> wrote:
>
>     Myriam:
>     Thank you very much for providing these results!
>     I have put effort to accelerate execution time and avoid using
>     global sizes in PtAP, for which the algorithm of transpose of
>     P_local and P_other likely doubles the memory usage. I'll try to
>     investigate why it becomes unscalable.
>     Hong
>
>         Hi,
>
>         you'll find the new scaling attached (green line). I used the
>         version 3.11 and the four scalability options :
>         -matptap_via scalable
>         -inner_diag_matmatmult_via scalable
>         -inner_offdiag_matmatmult_via scalable
>         -mat_freeintermediatedatastructures
>
>         The scaling is much better! The code even uses less memory for
>         the smallest cases. There is still an increase for the larger
>         one.
>
>         With regard to the time scaling, I used KSPView and LogView on
>         the two previous scalings (blue and yellow lines) but not on
>         the last one (green line). So we can't really compare them, am
>         I right? However, we can see that the new time scaling looks
>         quite good. It slightly increases from ~8s to ~27s.
>
>         Unfortunately, the computations are expensive so I would like
>         to avoid re-run them if possible. How relevant would be a
>         proper time scaling for you??
>
>         Myriam
>
>
>         Le 04/12/19 ? 18:18, Zhang, Hong a ?crit?:
>>         Myriam :
>>         Thanks for your effort. It will help us improve PETSc.
>>         Hong
>>
>>             Hi all,
>>
>>             I used the wrong script, that's why it diverged... Sorry
>>             about that.?
>>             I tried again with the right script applied on a tiny
>>             problem (~200
>>             elements). I can see a small difference in memory usage
>>             (gain ~ 1mB).
>>             when adding the -mat_freeintermediatestructures option. I
>>             still have to
>>             execute larger cases to plot the scaling. The
>>             supercomputer I am used to
>>             run my jobs on is really busy at the moment so it takes a
>>             while. I hope
>>             I'll send you the results on Monday.
>>
>>             Thanks everyone,
>>
>>             Myriam
>>
>>
>>             Le 04/11/19 ? 06:01, Jed Brown a ?crit?:
>>             > "Zhang, Hong" <hzhang at mcs.anl.gov
>>             <mailto:hzhang at mcs.anl.gov>> writes:
>>             >
>>             >> Jed:
>>             >>>> Myriam,
>>             >>>> Thanks for the plot.
>>             '-mat_freeintermediatedatastructures' should not affect
>>             solution. It releases almost half of memory in C=PtAP if
>>             C is not reused.
>>             >>> And yet if turning it on causes divergence, that
>>             would imply a bug.
>>             >>> Hong, are you able to reproduce the experiment to see
>>             the memory
>>             >>> scaling?
>>             >> I like to test his code using an alcf machine, but my
>>             hands are full now. I'll try it as soon as I find time,
>>             hopefully next week.
>>             > I have now compiled and run her code locally.
>>             >
>>             > Myriam, thanks for your last mail adding configuration
>>             and removing the
>>             > MemManager.h dependency.? I ran with and without
>>             > -mat_freeintermediatedatastructures and don't see a
>>             difference in
>>             > convergence.? What commands did you run to observe that
>>             difference?
>>
>>             -- 
>>             Myriam Peyrounette
>>             CNRS/IDRIS - HLST
>>             --
>>
>>
>
>         -- 
>         Myriam Peyrounette
>         CNRS/IDRIS - HLST
>         --
>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/9c78f543/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/9c78f543/attachment-0001.p7s>

From mfadams at lbl.gov  Tue Apr 30 07:43:57 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Apr 2019 08:43:57 -0400
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4GkPNv4k501_+CpB2yXtxV812PgJLxPLKkvM3zx=3_mLjg@mail.gmail.com>
	<fb8151ff-aadd-bb5e-1f1d-b40a5565f90b@idris.fr>
	<CAMYG4GkXCeQ0tVOgjBk7QjqXqOFVYkngxTmG7gF8S27LRgH0=Q@mail.gmail.com>
Message-ID: <CADOhEh5ep_DTkub_VV+K=6hdKsLBA1L1B7s76NyinVT3c9Dwxg@mail.gmail.com>

On Tue, Mar 5, 2019 at 8:06 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette <
> myriam.peyrounette at idris.fr> wrote:
>
>> Hi Matt,
>>
>> I plotted the memory scalings using different threshold values. The two
>> scalings are slightly translated (from -22 to -88 mB) but this gain is
>> neglectable. The 3.6-scaling keeps being robust while the 3.10-scaling
>> deteriorates.
>>
>> Do you have any other suggestion?
>>
>> Mark, what is the option she can give to output all the GAMG data?
>

I think we did this and it was fine, or am I getting threads mixed up?

Use -info and grep on GAMG. This will print out the average nnz/row on each
level, which is way of seeing if the coarse grids are getting out of
control. But coarse grids are smaller so they should not be a big deal.


>
> Also, run using -ksp_view. GAMG will report all the sizes of its grids, so
> it should be easy to see
> if the coarse grid sizes are increasing, and also what the effect of the
> threshold value is.
>

The coarse grid "sizes" will go way down, that is what MG does unless
something is very wrong. The nnz/row will go up in many cases. If ksp_view
prints out the nnz for the operator on each level then you can compute the
average nnz/row (-info just does that for you).

The only change that I can think of in GAMG wrt coarsening was the
treatment of the square graph parameters. It used to be a bool (square
first level or not). Now it is an integer q (square the first q levels).

If you suspect GAMG, can you test with Hypre?


>
>   Thanks,
>
>      Matt
>
>> Thanks
>> Myriam
>>
>> Le 03/02/19 ? 02:27, Matthew Knepley a ?crit :
>>
>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
>>> to 3.10, this code has a bad memory scaling.
>>>
>>> To report this issue, I took the PETSc script ex42.c and slightly
>>> modified it so that the KSP and PC configurations are the same as in my
>>> code. In particular, I use a "personnalised" multi-grid method. The
>>> modifications are indicated by the keyword "TopBridge" in the attached
>>> scripts.
>>>
>>> To plot the memory (weak) scaling, I ran four calculations for each
>>> script with increasing problem sizes and computations cores:
>>>
>>> 1. 100,000 elts on 4 cores
>>> 2. 1 million elts on 40 cores
>>> 3. 10 millions elts on 400 cores
>>> 4. 100 millions elts on 4,000 cores
>>>
>>> The resulting graph is also attached. The scaling using PETSc 3.10
>>> clearly deteriorates for large cases, while the one using PETSc 3.6 is
>>> robust.
>>>
>>> After a few tests, I found that the scaling is mostly sensitive to the
>>> use of the AMG method for the coarse grid (line 1780 in
>>> main_ex42_petsc36.cc). In particular, the performance strongly
>>> deteriorates when commenting lines 1777 to 1790 (in
>>> main_ex42_petsc36.cc).
>>>
>>> Do you have any idea of what changed between version 3.6 and version
>>> 3.10 that may imply such degradation?
>>>
>>
>> I believe the default values for PCGAMG changed between versions. It
>> sounds like the coarsening rate
>> is not great enough, so that these grids are too large. This can be set
>> using:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>
>> There is some explanation of this effect on that page. Let us know if
>> setting this does not correct the situation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Let me know if you need further information.
>>>
>>> Best,
>>>
>>> Myriam Peyrounette
>>>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/575688b1/attachment.html>

From boyceg at gmail.com  Tue Apr 30 07:53:13 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 08:53:13 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
Message-ID: <CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>



> On Apr 29, 2019, at 9:47 PM, Boyce Griffith <boyceg at gmail.com> wrote:
> 
> 
> 
>> On Apr 29, 2019, at 8:39 PM, Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> 
>> I believe so. Probably you've already done this but you can look at src/ksp/ksp/examples/tutorials ex62.c and ex64.c for sample usage.
> 
> Yep!
> 
>> Do you get the same (correct) behavior for both when you use inner_ises but no outer (let GASM/ASM determine it)? What if you use the same outer as inner?
> 
> I think that I had a bug in setting up the overlapping subdomains. Now GASM seems to work, but the convergence history does not seem to depend on how I set the outer IS, which seems surprising. In particular, with GASM, I?ve tried setting the outer IS to include overlaps; setting the outer IS to be the inner IS; and setting the outer IS to NULL ? all converge in 2 iterations using exact subdomain solves. (This is a problem with only 2 subdomains.)
> 
> ASM gives different results. Specifically, if I let ASM determine the overlaps automatically, then it converges in 4 iterations. If I use the inner IS and the overlapping IS, then it takes nearly 40.

Ah, I may have spoken too soon.

Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.

Allowing ASM to construct the "outer" subdomains from the same non-overlapping "inner" subdomains used with GASM, and using "exact" subdomain solvers, I get convergence in ~4 iterations in both the preconditioned and true residual norm to an rtol of 1e-12.

>> Note that for ASM the "inner" subdomains can overlap while for GASM they cannot overlap (and of course their union needs to be all degrees of freedom).
> 
> I think that I have been setting up the inner subdomains correctly for GASM. I have some doubts about how I am setting up the overlapping regions, and so I think I will just let GASM setup the overlaps for now.
> 
>> GASM is usually used where each inner is exactly several processes but it should work more generally as well.
> 
> This is a situation where I have different PDEs on two subdomains with a flux continuity condition along the interface. One subdomain has a scalar PDE (just Laplace?s equation), the other has a vector PDE. I am trying to use GASM to split up the problems, and then to use GAMG on the scalar domain and FIELDSPLIT on the vector domain. I already know that FIELDSPLIT can be made to work well on the vector PDE.

BTW, is GASM the right thing to be using for this kind of domain decomposition?

Thanks!

-- Boyce

> Thanks!
> 
> ? Boyce
> 
>>  Barry
>> 
>> 
>>> On Apr 29, 2019, at 7:10 PM, Griffith, Boyce Eugene via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> I am trying to setup a GASM preconditioner and am running into some difficulty getting convergence even with exact subdomain solvers. I am just doing things in serial now, and tried switching to ASM, and get convergence. I am wondering if I am misunderstanding the GASM interface. If inner_ises and outer_ises define the ?inner? and ?outer? subdomains for GASM, should GASM and ASM do the same things when configured via:
>>> 
>>> 	PCGASMSetSubdomains(pc, n_subdomains, inner_ises, outer_ises);
>>> 
>>> and
>>> 
>>> 	PCASMSetLocalSubdomains(pc, n_subdomains, outer_ises, inner_ises);
>>> 
>>> Thanks!
>>> 
>>> ? Boyce
>> 
> 


From mfadams at lbl.gov  Tue Apr 30 08:06:51 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Apr 2019 09:06:51 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
Message-ID: <CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>

>
>
>
>
> Allowing GASM to construct the "outer" subdomains from the non-overlapping
> "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are
> using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations
> in the preconditioned residual norm to an rtol of 1e-12. If I look at the
> true residual norm, however, it stagnates around 1e-4.
>
>
That PC is singular.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/79f66d78/attachment.html>

From boyceg at gmail.com  Tue Apr 30 08:38:55 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 09:38:55 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
Message-ID: <E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>



> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> 
> 
> 
> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
> 
> 
> That PC is singular. 


Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.

One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html>), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html>). However, changing the GASM overlap does not make any difference in the convergence history.

-- Boyce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/dea41e50/attachment-0001.html>

From fdkong.jd at gmail.com  Tue Apr 30 10:00:56 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 30 Apr 2019 09:00:56 -0600
Subject: [petsc-users] Bad memory scaling with PETSc 3.10
In-Reply-To: <31946231-4948-fc6a-093a-7ed8f00f3579@idris.fr>
References: <afa265c1-1678-2ad7-bb99-0af540a75931@idris.fr>
	<CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com>
	<0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr>
	<00dfa074-cc41-6a3b-257d-28a089e90617@idris.fr>
	<877ecaeo63.fsf@jedbrown.org>
	<9629bc17-89d4-fdc9-1b44-7c4a25f62861@idris.fr>
	<CAGCphBvr6kTE7YiGYMNeSsORJiZ1RLOym+BDMXQBUvNk99OhGw@mail.gmail.com>
	<3ad6ca66-d665-739a-874e-7599d5270797@idris.fr>
	<CADOhEh6v-SSXPG_KJ=_+DJZ8pAL=Hw2BGXASue_GRLTdBWpbBA@mail.gmail.com>
	<CAGCphBtbC3jZ+UPxWQyfVfufSrXH6z6nU=0P_CjsWYLbSpCebg@mail.gmail.com>
	<877ec17g52.fsf@jedbrown.org>
	<CAGCphBuayVg=VKKhs2_5gyaeyvYqTzdp6F=Rb6ADu6kMApYDEg@mail.gmail.com>
	<87zhox5gxi.fsf@jedbrown.org>
	<93a32d83-0b81-8bf3-d654-6711d9b0138f@idris.fr>
	<CAGCphBsSuQJj96emb4vjkZZ8+c991B9WVcPLvi9N+mmbHSdnpg@mail.gmail.com>
	<26b73c92-6a23-cf03-9e7f-1a24893ee512@idris.fr>
	<CAGCphBvNdLMfZhrXDBtdgzaikujQu98WarT6Qr9LGF5i0AAqUA@mail.gmail.com>
	<CAGCphBs6Svo7vohR_x+MzHCwu_ycNW475s-F6P0PKH8xPzj0fg@mail.gmail.com>
	<31946231-4948-fc6a-093a-7ed8f00f3579@idris.fr>
Message-ID: <CAN5Wd-JAacDFEWmxbJ1PFY0gQWU5NJweEF=ctk+J-eCDv6BViA@mail.gmail.com>

HI Myriam,

We are interesting how the new algorithms perform. So there are two new
algorithms you could try.

Algorithm 1:

-matptap_via allatonce  -mat_freeintermediatedatastructures 1

Algorithm 2:

-matptap_via allatonce_merged -mat_freeintermediatedatastructures 1


Note that you need to use the current petsc-master, and also please put
"-snes_view" in your script so that we can confirm these options are
actually get set.

Thanks,

Fande,


On Tue, Apr 30, 2019 at 2:26 AM Myriam Peyrounette via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> that's really good news for us, thanks! I will plot again the memory
> scaling using these new options and let you know. Next week I hope.
>
> Before that, I just need to clarify the situation. Throughout our
> discussions, we mentionned a number of options concerning the scalability:
>
> -matptatp_via scalable
> -inner_diag_matmatmult_via scalable
> -inner_diag_matmatmult_via scalable
> -mat_freeintermediatedatastructures
> -matptap_via allatonce
> -matptap_via allatonce_merged
>
> Which ones of them are compatible? Should I use all of them at the same
> time? Is there redundancy?
>
> Thanks,
>
> Myriam
>
> Le 04/25/19 ? 21:47, Zhang, Hong a ?crit :
>
> Myriam:
> Checking MatPtAP() in petsc-3.6.4, I realized that it uses different
> algorithm than petsc-10 and later versions. petsc-3.6 uses out-product for
> C=P^T * AP, while petsc-3.10 uses local transpose of P. petsc-3.10
> accelerates data accessing, but doubles the memory of P.
>
> Fande added two new implementations for MatPtAP() to petsc-master which
> use much smaller and scalable memories with slightly higher computing time
> (faster than hypre though). You may use these new implementations if you
> have concern on memory scalability. The option for these new implementation
> are:
> -matptap_via allatonce
> -matptap_via allatonce_merged
>
> Hong
>
> On Mon, Apr 15, 2019 at 12:10 PM hzhang at mcs.anl.gov <hzhang at mcs.anl.gov>
> wrote:
>
>> Myriam:
>> Thank you very much for providing these results!
>> I have put effort to accelerate execution time and avoid using global
>> sizes in PtAP, for which the algorithm of transpose of P_local and P_other
>> likely doubles the memory usage. I'll try to investigate why it becomes
>> unscalable.
>> Hong
>>
>>> Hi,
>>>
>>> you'll find the new scaling attached (green line). I used the version
>>> 3.11 and the four scalability options :
>>> -matptap_via scalable
>>> -inner_diag_matmatmult_via scalable
>>> -inner_offdiag_matmatmult_via scalable
>>> -mat_freeintermediatedatastructures
>>>
>>> The scaling is much better! The code even uses less memory for the
>>> smallest cases. There is still an increase for the larger one.
>>>
>>> With regard to the time scaling, I used KSPView and LogView on the two
>>> previous scalings (blue and yellow lines) but not on the last one (green
>>> line). So we can't really compare them, am I right? However, we can see
>>> that the new time scaling looks quite good. It slightly increases from ~8s
>>> to ~27s.
>>>
>>> Unfortunately, the computations are expensive so I would like to avoid
>>> re-run them if possible. How relevant would be a proper time scaling for
>>> you?
>>>
>>> Myriam
>>>
>>> Le 04/12/19 ? 18:18, Zhang, Hong a ?crit :
>>>
>>> Myriam :
>>> Thanks for your effort. It will help us improve PETSc.
>>> Hong
>>>
>>> Hi all,
>>>>
>>>> I used the wrong script, that's why it diverged... Sorry about that.
>>>> I tried again with the right script applied on a tiny problem (~200
>>>> elements). I can see a small difference in memory usage (gain ~ 1mB).
>>>> when adding the -mat_freeintermediatestructures option. I still have to
>>>> execute larger cases to plot the scaling. The supercomputer I am used to
>>>> run my jobs on is really busy at the moment so it takes a while. I hope
>>>> I'll send you the results on Monday.
>>>>
>>>> Thanks everyone,
>>>>
>>>> Myriam
>>>>
>>>>
>>>> Le 04/11/19 ? 06:01, Jed Brown a ?crit :
>>>> > "Zhang, Hong" <hzhang at mcs.anl.gov> writes:
>>>> >
>>>> >> Jed:
>>>> >>>> Myriam,
>>>> >>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should
>>>> not affect solution. It releases almost half of memory in C=PtAP if C is
>>>> not reused.
>>>> >>> And yet if turning it on causes divergence, that would imply a bug.
>>>> >>> Hong, are you able to reproduce the experiment to see the memory
>>>> >>> scaling?
>>>> >> I like to test his code using an alcf machine, but my hands are full
>>>> now. I'll try it as soon as I find time, hopefully next week.
>>>> > I have now compiled and run her code locally.
>>>> >
>>>> > Myriam, thanks for your last mail adding configuration and removing
>>>> the
>>>> > MemManager.h dependency.  I ran with and without
>>>> > -mat_freeintermediatedatastructures and don't see a difference in
>>>> > convergence.  What commands did you run to observe that difference?
>>>>
>>>> --
>>>> Myriam Peyrounette
>>>> CNRS/IDRIS - HLST
>>>> --
>>>>
>>>>
>>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/e99a7d77/attachment.html>

From fdkong.jd at gmail.com  Tue Apr 30 10:23:12 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 30 Apr 2019 09:23:12 -0600
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
Message-ID: <CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>

On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
>
>>
>>
>> Allowing GASM to construct the "outer" subdomains from the
>> non-overlapping "inner" subdomains, and using "exact" subdomain solvers
>> (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get
>> convergence in ~2 iterations in the preconditioned residual norm to an rtol
>> of 1e-12. If I look at the true residual norm, however, it stagnates around
>> 1e-4.
>>
>>
> That PC is singular.
>
>
> Yes. I am confused about why GASM is giving a singular PC but ASM is not
> with the same subdomains.
>

We could not tell much without the info of detailed solver setup. The
overlapping function was implemented by me and Dmitry a couple of years
ago, and it is trick when a subdomain is shared by multiple cores.  Do you
mind to start from our example? Or setup an example for us to demonstrate
your issues?

At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear
solver) to print more information that will help us a bit.




>
> One difference between them is that the default GASM overlap is 0 (
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html),
> but the default ASM overlap is 1 (
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html
> ).
>

For your particular application, not sure you need any overlap since there
are two different PDEs on two different subdomains. It may be fine to run
the PC without the overlapping domain.   It is a definitely interesting
application for GASM.

Fande,



> However, changing the GASM overlap does not make any difference in the
> convergence history.
>
> -- Boyce
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/2044522b/attachment.html>

From boyceg at gmail.com  Tue Apr 30 10:33:42 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 11:33:42 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
Message-ID: <24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>



> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> 
> 
> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> 
> 
>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> 
>> 
>> 
>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>> 
>> 
>> That PC is singular. 
> 
> 
> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
> 
> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
> 
> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.

Here you go:

$ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
make: `main' is up to date.
Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view

  0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
  2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
  3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
  4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
  5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
  6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
  7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
  8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
  9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
Linear solve did not converge due to DIVERGED_ITS iterations 10
KSP Object: 1 MPI processes
  type: fgmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10, nonzero initial guess
  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gasm
    Restriction/interpolation type: RESTRICT
    requested amount of overlap = 0
    total number of subdomains = 2
    max number of local subdomains = 2
    [0|1] number of locally-supported subdomains = 2
    Subdomain solver info is as follows:
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
    KSP Object: (sub_) 1 MPI processes
      type: fgmres
        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        happy breakdown tolerance 1e-30
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
      right preconditioning
      using UNPRECONDITIONED norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=484, cols=484
              package used to perform factorization: petsc
              total: nonzeros=15376, allocated nonzeros=15376
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=484, cols=484
        total: nonzeros=15376, allocated nonzeros=15376
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
    KSP Object: (sub_) 1 MPI processes
      type: fgmres
        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        happy breakdown tolerance 1e-30
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
      right preconditioning
      using UNPRECONDITIONED norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=121, cols=121
              package used to perform factorization: petsc
              total: nonzeros=961, allocated nonzeros=961
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=121, cols=121
        total: nonzeros=961, allocated nonzeros=961
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
[0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
[0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
[0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
[0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
[0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
[0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
[0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c

> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html>), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html>).
> 
> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
> 
> Fande,
> 
>  
> However, changing the GASM overlap does not make any difference in the convergence history.
> 
> -- Boyce

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/a401d43f/attachment-0001.html>

From mfadams at lbl.gov  Tue Apr 30 11:31:05 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Apr 2019 12:31:05 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
Message-ID: <CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>

When I said it was singular I was looking at "preconditioned residual norm
to an rtol of 1e-12. If I look at the true residual norm, however, it
stagnates around 1e-4."

This is not what I am seeing in this output. It is just a poor PC. The big
drop in the residual at the beginning is suspicious. When you solve this
problem well have you checked that you are getting a good solution? That
is, do you check that your model is not messed up, like a bad mesh.

On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>
>
>
> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>
>>
>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>
>>>
>>>
>>> Allowing GASM to construct the "outer" subdomains from the
>>> non-overlapping "inner" subdomains, and using "exact" subdomain solvers
>>> (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get
>>> convergence in ~2 iterations in the preconditioned residual norm to an rtol
>>> of 1e-12. If I look at the true residual norm, however, it stagnates around
>>> 1e-4.
>>>
>>>
>> That PC is singular.
>>
>>
>> Yes. I am confused about why GASM is giving a singular PC but ASM is not
>> with the same subdomains.
>>
>
> We could not tell much without the info of detailed solver setup. The
> overlapping function was implemented by me and Dmitry a couple of years
> ago, and it is trick when a subdomain is shared by multiple cores.  Do you
> mind to start from our example? Or setup an example for us to demonstrate
> your issues?
>
> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear
> solver) to print more information that will help us a bit.
>
>
> Here you go:
>
> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu
> -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual
> -ksp_max_it 10 -ksp_view
> make: `main' is up to date.
> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres
> -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason
> -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>
>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm
> 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm
> 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>   2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm
> 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>   3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm
> 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>   4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm
> 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>   5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm
> 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>   6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm
> 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>   7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm
> 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>   8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm
> 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>   9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm
> 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>  10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm
> 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
> Linear solve did not converge due to DIVERGED_ITS iterations 10
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10, nonzero initial guess
>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gasm
>     Restriction/interpolation type: RESTRICT
>     requested amount of overlap = 0
>     total number of subdomains = 2
>     max number of local subdomains = 2
>     [0|1] number of locally-supported subdomains = 2
>     Subdomain solver info is as follows:
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>     KSP Object: (sub_) 1 MPI processes
>       type: fgmres
>         restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>       right preconditioning
>       using UNPRECONDITIONED norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=484, cols=484
>               package used to perform factorization: petsc
>               total: nonzeros=15376, allocated nonzeros=15376
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=484, cols=484
>         total: nonzeros=15376, allocated nonzeros=15376
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>     KSP Object: (sub_) 1 MPI processes
>       type: fgmres
>         restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>       right preconditioning
>       using UNPRECONDITIONED norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=121, cols=121
>               package used to perform factorization: petsc
>               total: nonzeros=961, allocated nonzeros=961
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=121, cols=121
>         total: nonzeros=961, allocated nonzeros=961
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
> *[0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------*
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown
> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu
> by boyceg Tue Apr 30 11:29:02 2019
> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90
> --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1
> --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
> [0]PETSC ERROR: #2 PCView_GASM() line 251 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
> [0]PETSC ERROR: #3 PCView() line 1651 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPView() line 213 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
> [0]PETSC ERROR: #5 PetscObjectView() line 100 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
> [0]PETSC ERROR: #6 ObjectView() line 14 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #7 KSPSolve() line 831 in
> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>
> One difference between them is that the default GASM overlap is 0 (
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html),
>> but the default ASM overlap is 1 (
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html
>> ).
>>
>
> For your particular application, not sure you need any overlap since there
> are two different PDEs on two different subdomains. It may be fine to run
> the PC without the overlapping domain.   It is a definitely interesting
> application for GASM.
>
> Fande,
>
>
>
>> However, changing the GASM overlap does not make any difference in the
>> convergence history.
>>
>> -- Boyce
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/e505c820/attachment-0001.html>

From boyceg at gmail.com  Tue Apr 30 11:36:51 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 12:36:51 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
Message-ID: <1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>



> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
> 
> This is not what I am seeing in this output.

Whoops, I switched the outer KSP from GMRES to FGMRES.

With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):

$ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual

  0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
  2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
  3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
  4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
  5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
  6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
  7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
  8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
  9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04

> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?

Yes indeed, the solution appears to be no good with this preconditioner.

Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.

> That is, do you check that your model is not messed up, like a bad mesh.

This is a uniform HEX mesh automatically generated by libMesh.

> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> 
> 
>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com <mailto:fdkong.jd at gmail.com>> wrote:
>> 
>> 
>> 
>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> 
>> 
>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>>> 
>>> 
>>> 
>>> 
>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>> 
>>> 
>>> That PC is singular. 
>> 
>> 
>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>> 
>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>> 
>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
> 
> Here you go:
> 
> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
> make: `main' is up to date.
> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
> 
>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>   2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>   3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>   4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>   5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>   6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>   7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>   8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>   9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>  10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
> Linear solve did not converge due to DIVERGED_ITS iterations 10
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10, nonzero initial guess
>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gasm
>     Restriction/interpolation type: RESTRICT
>     requested amount of overlap = 0
>     total number of subdomains = 2
>     max number of local subdomains = 2
>     [0|1] number of locally-supported subdomains = 2
>     Subdomain solver info is as follows:
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>     KSP Object: (sub_) 1 MPI processes
>       type: fgmres
>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>       right preconditioning
>       using UNPRECONDITIONED norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=484, cols=484
>               package used to perform factorization: petsc
>               total: nonzeros=15376, allocated nonzeros=15376
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=484, cols=484
>         total: nonzeros=15376, allocated nonzeros=15376
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>     KSP Object: (sub_) 1 MPI processes
>       type: fgmres
>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>         happy breakdown tolerance 1e-30
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>       right preconditioning
>       using UNPRECONDITIONED norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=121, cols=121
>               package used to perform factorization: petsc
>               total: nonzeros=961, allocated nonzeros=961
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=121, cols=121
>         total: nonzeros=961, allocated nonzeros=961
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu <http://boycesimacwork.dhcp.unc.edu/> by boyceg Tue Apr 30 11:29:02 2019
> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
> 
>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html>), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html>).
>> 
>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>> 
>> Fande,
>> 
>>  
>> However, changing the GASM overlap does not make any difference in the convergence history.
>> 
>> -- Boyce
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/dcb5b291/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Tue Apr 30 11:41:13 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Tue, 30 Apr 2019 11:41:13 -0500
Subject: [petsc-users] Quick question about ISCreateGeneral
Message-ID: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>

Hi PETSc Developers,

I see that in the examples for ISCreateGeneral, the index sets are created
by copying values from int arrays (which were created by PetscMalloc1 which
is not collective).

If I the ISCreateGeneral is called with PETSC_COMM_WORLD and the int arrays
on each rank are independently created, does the index set created
concatenate all the int-arrays into one ? If not, what needs to be done to
get such an index set ?

PS: For context, I want to write a fftshift convenience function (like
numpy, MATLAB) but for large distributed vectors. I thought that I could do
this with VecScatter and two index sets, one shifted and one un-shifted.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/9c8e5f6e/attachment.html>

From ysjosh.lo at gmail.com  Tue Apr 30 12:18:49 2019
From: ysjosh.lo at gmail.com (Josh L)
Date: Tue, 30 Apr 2019 12:18:49 -0500
Subject: [petsc-users] GlobalToLocal between DM and SubDM
Message-ID: <CA+2ebbDqC40Zw__apgdwzmzN2nvFC9vnE8dwr9pC869OVa6SKA@mail.gmail.com>

Hi,

I have a DM that that has 2 sections:

section_0 has 2 dofs: u, v
section_1 has 1 dofs: p

and a SubDM created by DMCreateSubDM. SubDM only has section_1.

There are a global vector created with DM, and a local vector created with
SubDM

Can I do GlobalToLocal between this 2 vectors? If yes, which DM should I
use when calling GlobalToLocal.

Basically, I want
a global vec: [u1, v1, p1 u2, v2, p2, u3, v3, p3......]
to
local vec: [all dof p that belongs to this processor]


Thanks,
Yu-Sheng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/1cbc5e63/attachment.html>

From knepley at gmail.com  Tue Apr 30 12:57:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 30 Apr 2019 13:57:04 -0400
Subject: [petsc-users] GlobalToLocal between DM and SubDM
In-Reply-To: <CA+2ebbDqC40Zw__apgdwzmzN2nvFC9vnE8dwr9pC869OVa6SKA@mail.gmail.com>
References: <CA+2ebbDqC40Zw__apgdwzmzN2nvFC9vnE8dwr9pC869OVa6SKA@mail.gmail.com>
Message-ID: <CAMYG4G=+YcN19eh2sSdtSisYf5W5uBUOq6AS_eOZ7jNnQvvCmQ@mail.gmail.com>

On Tue, Apr 30, 2019 at 1:20 PM Josh L via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I have a DM that that has 2 sections:
>
> section_0 has 2 dofs: u, v
> section_1 has 1 dofs: p
>
> and a SubDM created by DMCreateSubDM. SubDM only has section_1.
>
> There are a global vector created with DM, and a local vector created with
> SubDM
>
> Can I do GlobalToLocal between this 2 vectors?
>

No, the global and local vector have to have the same DM.

If yes, which DM should I use when calling GlobalToLocal.
>
> Basically, I want
> a global vec: [u1, v1, p1 u2, v2, p2, u3, v3, p3......]
> to
> local vec: [all dof p that belongs to this processor]
>

You will have to do it in two steps, either,

  1) Extract p in a subDM, and then do G2L on that

or

  2) Do G2L on the full DM, and then extract p from the local vec

1) sounds better to me, but I do not know what your code is doing.

  Thanks,

    Matt


>
> Thanks,
> Yu-Sheng
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/d1a2e35c/attachment.html>

From jczhang at mcs.anl.gov  Tue Apr 30 13:00:29 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 30 Apr 2019 18:00:29 +0000
Subject: [petsc-users] Quick question about ISCreateGeneral
In-Reply-To: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>
References: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>
Message-ID: <CA+MQGp9eAwNp=vk-LUERKZKLgkjQXo5OD6Xqb_Mb2jJu0kLSdg@mail.gmail.com>



On Tue, Apr 30, 2019 at 11:42 AM Sajid Ali via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi PETSc Developers,

I see that in the examples for ISCreateGeneral, the index sets are created by copying values from int arrays (which were created by PetscMalloc1 which is not collective).

If I the ISCreateGeneral is called with PETSC_COMM_WORLD and the int arrays on each rank are independently created, does the index set created concatenate all the int-arrays into one ? If not, what needs to be done to get such an index set ?
From my understanding, they are independently created and not concatenated.  I like index sets created with PETSC_COMM_SELF. They are easy to understand.

PS: For context, I want to write a fftshift convenience function (like numpy, MATLAB) but for large distributed vectors. I thought that I could do this with VecScatter and two index sets, one shifted and one un-shifted.
To achieve this, index sets created with PETSC_COMM_SELF are enough. They just need to contain global indices to indicate the MPI vector to MPI vector scatter. You can think each process provides one piece of the scatter.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/54be1669/attachment.html>

From ysjosh.lo at gmail.com  Tue Apr 30 13:19:43 2019
From: ysjosh.lo at gmail.com (Josh L)
Date: Tue, 30 Apr 2019 13:19:43 -0500
Subject: [petsc-users] GlobalToLocal between DM and SubDM
In-Reply-To: <CAMYG4G=+YcN19eh2sSdtSisYf5W5uBUOq6AS_eOZ7jNnQvvCmQ@mail.gmail.com>
References: <CA+2ebbDqC40Zw__apgdwzmzN2nvFC9vnE8dwr9pC869OVa6SKA@mail.gmail.com>
	<CAMYG4G=+YcN19eh2sSdtSisYf5W5uBUOq6AS_eOZ7jNnQvvCmQ@mail.gmail.com>
Message-ID: <CA+2ebbAVh67gD2pNMQ5bT-y-w-69+RKjK3xGDG3dA-sOvveppQ@mail.gmail.com>

Thank you.
I just have a coupled problem, and i want to solve it beginning with
stagger then couple.



Matthew Knepley <knepley at gmail.com> ? 2019?4?30? ?? ??12:57???

> On Tue, Apr 30, 2019 at 1:20 PM Josh L via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> I have a DM that that has 2 sections:
>>
>> section_0 has 2 dofs: u, v
>> section_1 has 1 dofs: p
>>
>> and a SubDM created by DMCreateSubDM. SubDM only has section_1.
>>
>> There are a global vector created with DM, and a local vector created
>> with SubDM
>>
>> Can I do GlobalToLocal between this 2 vectors?
>>
>
> No, the global and local vector have to have the same DM.
>
> If yes, which DM should I use when calling GlobalToLocal.
>>
>> Basically, I want
>> a global vec: [u1, v1, p1 u2, v2, p2, u3, v3, p3......]
>> to
>> local vec: [all dof p that belongs to this processor]
>>
>
> You will have to do it in two steps, either,
>
>   1) Extract p in a subDM, and then do G2L on that
>
> or
>
>   2) Do G2L on the full DM, and then extract p from the local vec
>
> 1) sounds better to me, but I do not know what your code is doing.
>
>   Thanks,
>
>     Matt
>
>
>>
>> Thanks,
>> Yu-Sheng
>>
>>
>>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/5aac3fa6/attachment.html>

From mfadams at lbl.gov  Tue Apr 30 13:25:13 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Apr 2019 14:25:13 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
Message-ID: <CADOhEh7HQZnfXfonpoCAe40JEoFsbkGGsqr0y6+Vn15nZg1xpg@mail.gmail.com>

My question about the quality of the solution was to check if the model
(eg, mesh) was messed up, not if the algebraic error was acceptable. So
does an exact solution look OK. Using LU if you need to. If there is say a
singularity it will mess up the solver as well as give you a bad solution.


On Tue, Apr 30, 2019 at 12:36 PM Boyce Griffith <boyceg at gmail.com> wrote:

>
>
> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
> When I said it was singular I was looking at "preconditioned residual norm
> to an rtol of 1e-12. If I look at the true residual norm, however, it
> stagnates around 1e-4."
>
> This is not what I am seeing in this output.
>
>
> Whoops, I switched the outer KSP from GMRES to FGMRES.
>
> With GMRES, the preconditioned residual norm drops nicely (but, of course,
> the solution is no good):
>
> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>
>   0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm
> 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm
> 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>   2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm
> 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>   3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm
> 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>   4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm
> 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>   5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm
> 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>   6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm
> 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>   7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm
> 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>   8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm
> 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>   9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm
> 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>  10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm
> 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>  11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm
> 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
>
> It is just a poor PC. The big drop in the residual at the beginning is
> suspicious. When you solve this problem well have you checked that you are
> getting a good solution?
>
>
> Yes indeed, the solution appears to be no good with this preconditioner.
>
> Note that all that the PC is doing is applying GASM to split the problem
> onto two subdomains.
>
> That is, do you check that your model is not messed up, like a bad mesh.
>
>
> This is a uniform HEX mesh automatically generated by libMesh.
>
> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>
>>
>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>
>>
>>
>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>>
>>>
>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>
>>>>
>>>>
>>>> Allowing GASM to construct the "outer" subdomains from the
>>>> non-overlapping "inner" subdomains, and using "exact" subdomain solvers
>>>> (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get
>>>> convergence in ~2 iterations in the preconditioned residual norm to an rtol
>>>> of 1e-12. If I look at the true residual norm, however, it stagnates around
>>>> 1e-4.
>>>>
>>>>
>>> That PC is singular.
>>>
>>>
>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not
>>> with the same subdomains.
>>>
>>
>> We could not tell much without the info of detailed solver setup. The
>> overlapping function was implemented by me and Dmitry a couple of years
>> ago, and it is trick when a subdomain is shared by multiple cores.  Do you
>> mind to start from our example? Or setup an example for us to demonstrate
>> your issues?
>>
>> At least please use "-ksp_view" (linear solver) or "-snes_view"
>> (nonlinear solver) to print more information that will help us a bit.
>>
>>
>> Here you go:
>>
>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type
>> ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual
>> -ksp_max_it 10 -ksp_view
>> make: `main' is up to date.
>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres
>> -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason
>> -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>
>>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm
>> 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm
>> 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>   2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm
>> 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>   3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm
>> 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>   4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm
>> 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>   5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm
>> 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>   6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm
>> 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>   7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm
>> 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>   8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm
>> 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>   9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm
>> 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>  10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm
>> 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>> KSP Object: 1 MPI processes
>>   type: fgmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=10, nonzero initial guess
>>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>   right preconditioning
>>   using UNPRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: gasm
>>     Restriction/interpolation type: RESTRICT
>>     requested amount of overlap = 0
>>     total number of subdomains = 2
>>     max number of local subdomains = 2
>>     [0|1] number of locally-supported subdomains = 2
>>     Subdomain solver info is as follows:
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=484, cols=484
>>               package used to perform factorization: petsc
>>               total: nonzeros=15376, allocated nonzeros=15376
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=484, cols=484
>>         total: nonzeros=15376, allocated nonzeros=15376
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=121, cols=121
>>               package used to perform factorization: petsc
>>               total: nonzeros=961, allocated nonzeros=961
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=121, cols=121
>>         total: nonzeros=961, allocated nonzeros=961
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>> *[0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------*
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown
>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu
>> by boyceg Tue Apr 30 11:29:02 2019
>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90
>> --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1
>> --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>> [0]PETSC ERROR: #3 PCView() line 1651 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #4 KSPView() line 213 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>> [0]PETSC ERROR: #6 ObjectView() line 14 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #7 KSPSolve() line 831 in
>> /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>
>> One difference between them is that the default GASM overlap is 0 (
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html),
>>> but the default ASM overlap is 1 (
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html
>>> ).
>>>
>>
>> For your particular application, not sure you need any overlap since
>> there are two different PDEs on two different subdomains. It may be fine to
>> run the PC without the overlapping domain.   It is a definitely interesting
>> application for GASM.
>>
>> Fande,
>>
>>
>>
>>> However, changing the GASM overlap does not make any difference in the
>>> convergence history.
>>>
>>> -- Boyce
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/a9c5b16f/attachment-0001.html>

From boyceg at gmail.com  Tue Apr 30 13:34:52 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 14:34:52 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <CADOhEh7HQZnfXfonpoCAe40JEoFsbkGGsqr0y6+Vn15nZg1xpg@mail.gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
	<CADOhEh7HQZnfXfonpoCAe40JEoFsbkGGsqr0y6+Vn15nZg1xpg@mail.gmail.com>
Message-ID: <39BEC9C3-43A3-4874-86D1-8B20D1A87A5A@gmail.com>



> On Apr 30, 2019, at 2:25 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> My question about the quality of the solution was to check if the model (eg, mesh) was messed up, not if the algebraic error was acceptable. So does an exact solution look OK. Using LU if you need to. If there is say a singularity it will mess up the solver as well as give you a bad solution.

 I appear to get consistent and reasonable looking solutions if I use ILU or FIELDSPLIT+GAMG for the preconditioner.

> On Tue, Apr 30, 2019 at 12:36 PM Boyce Griffith <boyceg at gmail.com <mailto:boyceg at gmail.com>> wrote:
> 
> 
>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>> 
>> This is not what I am seeing in this output.
> 
> Whoops, I switched the outer KSP from GMRES to FGMRES.
> 
> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
> 
> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
> 
>   0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>   2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>   3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>   4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>   5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>   6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>   7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>   8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>   9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>  10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>  11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
> 
>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
> 
> Yes indeed, the solution appears to be no good with this preconditioner.
> 
> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
> 
>> That is, do you check that your model is not messed up, like a bad mesh.
> 
> This is a uniform HEX mesh automatically generated by libMesh.
> 
>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> 
>> 
>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com <mailto:fdkong.jd at gmail.com>> wrote:
>>> 
>>> 
>>> 
>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>> 
>>> 
>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>> 
>>>> 
>>>> That PC is singular. 
>>> 
>>> 
>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>> 
>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>> 
>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>> 
>> Here you go:
>> 
>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>> make: `main' is up to date.
>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>> 
>>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>   2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>   3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>   4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>   5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>   6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>   7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>   8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>   9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>  10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>> KSP Object: 1 MPI processes
>>   type: fgmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=10, nonzero initial guess
>>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>   right preconditioning
>>   using UNPRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: gasm
>>     Restriction/interpolation type: RESTRICT
>>     requested amount of overlap = 0
>>     total number of subdomains = 2
>>     max number of local subdomains = 2
>>     [0|1] number of locally-supported subdomains = 2
>>     Subdomain solver info is as follows:
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=484, cols=484
>>               package used to perform factorization: petsc
>>               total: nonzeros=15376, allocated nonzeros=15376
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=484, cols=484
>>         total: nonzeros=15376, allocated nonzeros=15376
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=121, cols=121
>>               package used to perform factorization: petsc
>>               total: nonzeros=961, allocated nonzeros=961
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=121, cols=121
>>         total: nonzeros=961, allocated nonzeros=961
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu <http://boycesimacwork.dhcp.unc.edu/> by boyceg Tue Apr 30 11:29:02 2019
>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>> 
>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html>), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html>).
>>> 
>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>> 
>>> Fande,
>>> 
>>>  
>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>> 
>>> -- Boyce
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/8c43a722/attachment-0001.html>

From knepley at gmail.com  Tue Apr 30 16:04:00 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 30 Apr 2019 17:04:00 -0400
Subject: [petsc-users] Quick question about ISCreateGeneral
In-Reply-To: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>
References: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>
Message-ID: <CAMYG4G=xS44g5tun4pD=zyN=TbR_62GmnjUdJwf=iz=WbmAWvw@mail.gmail.com>

On Tue, Apr 30, 2019 at 12:42 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi PETSc Developers,
>
> I see that in the examples for ISCreateGeneral, the index sets are created
> by copying values from int arrays (which were created by PetscMalloc1 which
> is not collective).
>
> If I the ISCreateGeneral is called with PETSC_COMM_WORLD and the int
> arrays on each rank are independently created, does the index set created
> concatenate all the int-arrays into one ? If not, what needs to be done to
> get such an index set ?
>

It does not sound scalable, but you can use
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISOnComm.html

   Matt


> PS: For context, I want to write a fftshift convenience function (like
> numpy, MATLAB) but for large distributed vectors. I thought that I could do
> this with VecScatter and two index sets, one shifted and one un-shifted.
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/54263004/attachment.html>

From bsmith at mcs.anl.gov  Tue Apr 30 16:21:13 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 30 Apr 2019 21:21:13 +0000
Subject: [petsc-users] Quick question about ISCreateGeneral
In-Reply-To: <CAMYG4G=xS44g5tun4pD=zyN=TbR_62GmnjUdJwf=iz=WbmAWvw@mail.gmail.com>
References: <CAOGsD9jmjygKHbJbwuMxBGcsOVW+s7kbKq6UW9rNVR5_vk5jrA@mail.gmail.com>
	<CAMYG4G=xS44g5tun4pD=zyN=TbR_62GmnjUdJwf=iz=WbmAWvw@mail.gmail.com>
Message-ID: <BD8681FA-3024-44C4-AD79-AA2157F1F3CB@anl.gov>


   Sajid,

     The comm in the IS isn't really important. It is the comm in the Vecs that matter.

 In the parallel vector to parallel vector case each process just provides "some" original locations (from) and their corresponding new locations (all in global numbering of the vector). Each process can provide any part of the "some".

 In the parallel to sequential vector or sequential to parallel vector case the indexing of the to (or from depending on which one is sequential) is relative to the local numbering of the sequential vector and each process can only list entries associated with its sequential vector.

  Barry


> On Apr 30, 2019, at 4:04 PM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Tue, Apr 30, 2019 at 12:42 PM Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi PETSc Developers, 
> 
> I see that in the examples for ISCreateGeneral, the index sets are created by copying values from int arrays (which were created by PetscMalloc1 which is not collective). 
> 
> If I the ISCreateGeneral is called with PETSC_COMM_WORLD and the int arrays on each rank are independently created, does the index set created concatenate all the int-arrays into one ? If not, what needs to be done to get such an index set ? 
> 
> It does not sound scalable, but you can use https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISOnComm.html
> 
>    Matt
>  
> PS: For context, I want to write a fftshift convenience function (like numpy, MATLAB) but for large distributed vectors. I thought that I could do this with VecScatter and two index sets, one shifted and one un-shifted.  
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From bsmith at mcs.anl.gov  Tue Apr 30 17:22:37 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 30 Apr 2019 22:22:37 +0000
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
Message-ID: <82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>


  Boyce,

    I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why! This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? 

> On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> 
>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>> 
>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>> 
>> This is not what I am seeing in this output.
> 
> Whoops, I switched the outer KSP from GMRES to FGMRES.
> 
> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
> 
> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
> 
>   0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>   2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>   3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>   4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>   5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>   6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>   7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>   8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>   9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>  10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>  11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
> 
>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
> 
> Yes indeed, the solution appears to be no good with this preconditioner.
> 
> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
> 
>> That is, do you check that your model is not messed up, like a bad mesh.
> 
> This is a uniform HEX mesh automatically generated by libMesh.
> 
>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> 
>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>> 
>>> 
>>> 
>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>> 
>>>> 
>>>> That PC is singular. 
>>> 
>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>> 
>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>> 
>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>> 
>> Here you go:
>> 
>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>> make: `main' is up to date.
>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>> 
>>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>   2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>   3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>   4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>   5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>   6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>   7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>   8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>   9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>  10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>> KSP Object: 1 MPI processes
>>   type: fgmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=10, nonzero initial guess
>>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>   right preconditioning
>>   using UNPRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: gasm
>>     Restriction/interpolation type: RESTRICT
>>     requested amount of overlap = 0
>>     total number of subdomains = 2
>>     max number of local subdomains = 2
>>     [0|1] number of locally-supported subdomains = 2
>>     Subdomain solver info is as follows:
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=484, cols=484
>>               package used to perform factorization: petsc
>>               total: nonzeros=15376, allocated nonzeros=15376
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=484, cols=484
>>         total: nonzeros=15376, allocated nonzeros=15376
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>     KSP Object: (sub_) 1 MPI processes
>>       type: fgmres
>>         restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>         happy breakdown tolerance 1e-30
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>       right preconditioning
>>       using UNPRECONDITIONED norm type for convergence test
>>     PC Object: (sub_) 1 MPI processes
>>       type: ilu
>>         out-of-place factorization
>>         0 levels of fill
>>         tolerance for zero pivot 2.22045e-14
>>         matrix ordering: natural
>>         factor fill ratio given 1., needed 1.
>>           Factored matrix follows:
>>             Mat Object: 1 MPI processes
>>               type: seqaij
>>               rows=121, cols=121
>>               package used to perform factorization: petsc
>>               total: nonzeros=961, allocated nonzeros=961
>>               total number of mallocs used during MatSetValues calls =0
>>                 not using I-node routines
>>       linear system matrix = precond matrix:
>>       Mat Object: () 1 MPI processes
>>         type: seqaij
>>         rows=121, cols=121
>>         total: nonzeros=961, allocated nonzeros=961
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>       - - - - - - - - - - - - - - - - - -
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>> 
>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html).
>>> 
>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>> 
>>> Fande,
>>> 
>>>  
>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>> 
>>> -- Boyce
>> 
> 


From boyceg at gmail.com  Tue Apr 30 22:33:38 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Tue, 30 Apr 2019 23:33:38 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
	<82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>
Message-ID: <7A10C5F8-7841-4D9F-A07B-0CCE85BC3DF6@gmail.com>


> On Apr 30, 2019, at 6:22 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Boyce,
> 
>    I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why!

Thanks for checking! I don?t know why either. I often use FGMRES for outer KSPs, and probably used it for the inner ones without thinking. However, switching to preonly does not appear to help here:

==========

Using GMRES as the outer KSP with ILU for the subdomain PC:

./main -d 3 -n 10 -ksp_type gmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view

  0 KSP preconditioned resid norm 7.956172961039e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.790924766864e-01 true resid norm 6.415624322140e+01 ||r(i)||/||b|| 1.222031831626e-01
  2 KSP preconditioned resid norm 1.021248567184e-02 true resid norm 1.531971188768e+00 ||r(i)||/||b|| 2.918059823653e-03
  3 KSP preconditioned resid norm 1.245290521533e-03 true resid norm 1.344228387002e-01 ||r(i)||/||b|| 2.560452101634e-04
  4 KSP preconditioned resid norm 1.090553693795e-04 true resid norm 1.055832655005e-01 ||r(i)||/||b|| 2.011123233687e-04
  5 KSP preconditioned resid norm 5.774209753120e-06 true resid norm 1.045883881261e-01 ||r(i)||/||b|| 1.992173062059e-04
  6 KSP preconditioned resid norm 4.856074176119e-07 true resid norm 1.047048288204e-01 ||r(i)||/||b|| 1.994390994840e-04
  7 KSP preconditioned resid norm 3.569834574103e-08 true resid norm 1.046981572164e-01 ||r(i)||/||b|| 1.994263915821e-04
  8 KSP preconditioned resid norm 1.914112077836e-09 true resid norm 1.046980716404e-01 ||r(i)||/||b|| 1.994262285791e-04
  9 KSP preconditioned resid norm 9.380275080687e-11 true resid norm 1.046980942278e-01 ||r(i)||/||b|| 1.994262716030e-04
 10 KSP preconditioned resid norm 3.490998066884e-12 true resid norm 1.046980929565e-01 ||r(i)||/||b|| 1.994262691815e-04
 11 KSP preconditioned resid norm 2.275544655754e-13 true resid norm 1.046980929905e-01 ||r(i)||/||b|| 1.994262692463e-04
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=20, nonzero initial guess
  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gasm
    Restriction/interpolation type: RESTRICT
    requested amount of overlap = 0
    total number of subdomains = 2
    max number of local subdomains = 2
    [0|1] number of locally-supported subdomains = 2
    Subdomain solver info is as follows:
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=484, cols=484
              package used to perform factorization: petsc
              total: nonzeros=15376, allocated nonzeros=15376
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=484, cols=484
        total: nonzeros=15376, allocated nonzeros=15376
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=121, cols=121
              package used to perform factorization: petsc
              total: nonzeros=961, allocated nonzeros=961
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=121, cols=121
        total: nonzeros=961, allocated nonzeros=961
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -

This shows convergence in the preconditioned norm but not in the true norm, and the solution appears nonphysical.

==========

Using GMRES for the outer KSP with LU for the subdomain PC:

./main -d 3 -n 10 -ksp_type gmres -sub_pc_type lu -ksp_monitor_true_residual -ksp_view

  0 KSP preconditioned resid norm 7.587523490344e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.452294080360e-15 true resid norm 1.026159563147e-01 ||r(i)||/||b|| 1.954602681716e-04
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=5000, nonzero initial guess
  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gasm
    Restriction/interpolation type: RESTRICT
    requested amount of overlap = 0
    total number of subdomains = 2
    max number of local subdomains = 2
    [0|1] number of locally-supported subdomains = 2
    Subdomain solver info is as follows:
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: lu
        out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: nd
        factor fill ratio given 5., needed 2.69979
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=484, cols=484
              package used to perform factorization: petsc
              total: nonzeros=41512, allocated nonzeros=41512
              total number of mallocs used during MatSetValues calls =0
                using I-node routines: found 194 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=484, cols=484
        total: nonzeros=15376, allocated nonzeros=15376
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: lu
        out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: nd
        factor fill ratio given 5., needed 2.57128
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=121, cols=121
              package used to perform factorization: petsc
              total: nonzeros=2471, allocated nonzeros=2471
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=121, cols=121
        total: nonzeros=961, allocated nonzeros=961
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -



==========

Using FGMRES as the outer KSP:

./main -d 3 -n 10 -ksp_type fgmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view

  0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 9.908062627364e+00 true resid norm 9.908062627364e+00 ||r(i)||/||b|| 1.887262612713e-02
  2 KSP unpreconditioned resid norm 1.198244246143e+00 true resid norm 1.198244246143e+00 ||r(i)||/||b|| 2.282385216660e-03
  3 KSP unpreconditioned resid norm 1.011712741080e-01 true resid norm 1.011712741080e-01 ||r(i)||/||b|| 1.927084741847e-04
  4 KSP unpreconditioned resid norm 9.890789240724e-02 true resid norm 9.890789240713e-02 ||r(i)||/||b|| 1.883972421881e-04
  5 KSP unpreconditioned resid norm 9.790912262081e-02 true resid norm 9.790912262142e-02 ||r(i)||/||b|| 1.864948108590e-04
  6 KSP unpreconditioned resid norm 9.758733533827e-02 true resid norm 9.758733534329e-02 ||r(i)||/||b|| 1.858818786218e-04
  7 KSP unpreconditioned resid norm 9.754239526505e-02 true resid norm 9.754239528968e-02 ||r(i)||/||b|| 1.857962779487e-04
  8 KSP unpreconditioned resid norm 9.732735871034e-02 true resid norm 9.732735925876e-02 ||r(i)||/||b|| 1.853866827767e-04
  9 KSP unpreconditioned resid norm 9.681037606662e-02 true resid norm 9.681036910553e-02 ||r(i)||/||b|| 1.844019330592e-04
 10 KSP unpreconditioned resid norm 9.672261014893e-02 true resid norm 9.672255386871e-02 ||r(i)||/||b|| 1.842346648258e-04
 11 KSP unpreconditioned resid norm 9.636675313899e-02 true resid norm 9.636858548398e-02 ||r(i)||/||b|| 1.835604348338e-04
 12 KSP unpreconditioned resid norm 9.626809760271e-02 true resid norm 9.624997274413e-02 ||r(i)||/||b|| 1.833345042985e-04
 13 KSP unpreconditioned resid norm 9.538127889520e-02 true resid norm 9.646040937528e-02 ||r(i)||/||b|| 1.837353386505e-04
 14 KSP unpreconditioned resid norm 7.698037210447e-02 true resid norm 1.004972547621e-01 ||r(i)||/||b|| 1.914246192480e-04
 15 KSP unpreconditioned resid norm 6.963471194663e-02 true resid norm 1.021051124617e-01 ||r(i)||/||b|| 1.944872257707e-04
 16 KSP unpreconditioned resid norm 6.288795988348e-02 true resid norm 1.048077408458e-01 ||r(i)||/||b|| 1.996351236971e-04
 17 KSP unpreconditioned resid norm 5.576252453977e-02 true resid norm 1.071287580389e-01 ||r(i)||/||b|| 2.040561383159e-04
 18 KSP unpreconditioned resid norm 5.096717927451e-02 true resid norm 1.092140137471e-01 ||r(i)||/||b|| 2.080280804443e-04
 19 KSP unpreconditioned resid norm 4.698350635435e-02 true resid norm 1.104283980783e-01 ||r(i)||/||b|| 2.103412088852e-04
 20 KSP unpreconditioned resid norm 4.387914680239e-02 true resid norm 1.116698560164e-01 ||r(i)||/||b|| 2.127059064451e-04
KSP Object: 1 MPI processes
  type: fgmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=20, nonzero initial guess
  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gasm
    Restriction/interpolation type: RESTRICT
    requested amount of overlap = 0
    total number of subdomains = 2
    max number of local subdomains = 2
    [0|1] number of locally-supported subdomains = 2
    Subdomain solver info is as follows:
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=484, cols=484
              package used to perform factorization: petsc
              total: nonzeros=15376, allocated nonzeros=15376
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=484, cols=484
        total: nonzeros=15376, allocated nonzeros=15376
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -
      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
    KSP Object: (sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (sub_) 1 MPI processes
      type: ilu
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 1., needed 1.
          Factored matrix follows:
            Mat Object: 1 MPI processes
              type: seqaij
              rows=121, cols=121
              package used to perform factorization: petsc
              total: nonzeros=961, allocated nonzeros=961
              total number of mallocs used during MatSetValues calls =0
                not using I-node routines
      linear system matrix = precond matrix:
      Mat Object: () 1 MPI processes
        type: seqaij
        rows=121, cols=121
        total: nonzeros=961, allocated nonzeros=961
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
      - - - - - - - - - - - - - - - - - -

==========

Again, note that using the same subdomains, ASM appears to work fine.

> This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? 
> 
>> On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> 
>> 
>>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>> 
>>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>>> 
>>> This is not what I am seeing in this output.
>> 
>> Whoops, I switched the outer KSP from GMRES to FGMRES.
>> 
>> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
>> 
>> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>> 
>>  0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>>  2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>>  3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>>  4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>>  5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>>  6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>>  7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>>  8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>>  9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>> 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>> 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
>> 
>>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
>> 
>> Yes indeed, the solution appears to be no good with this preconditioner.
>> 
>> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
>> 
>>> That is, do you check that your model is not messed up, like a bad mesh.
>> 
>> This is a uniform HEX mesh automatically generated by libMesh.
>> 
>>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>>> 
>>>>> 
>>>>> That PC is singular. 
>>>> 
>>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>>> 
>>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>>> 
>>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>>> 
>>> Here you go:
>>> 
>>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>> make: `main' is up to date.
>>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>> 
>>>  0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>>  2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>>  3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>>  4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>>  5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>>  6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>>  7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>>  8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>>  9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>> 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>>> KSP Object: 1 MPI processes
>>>  type: fgmres
>>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>    happy breakdown tolerance 1e-30
>>>  maximum iterations=10, nonzero initial guess
>>>  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>  right preconditioning
>>>  using UNPRECONDITIONED norm type for convergence test
>>> PC Object: 1 MPI processes
>>>  type: gasm
>>>    Restriction/interpolation type: RESTRICT
>>>    requested amount of overlap = 0
>>>    total number of subdomains = 2
>>>    max number of local subdomains = 2
>>>    [0|1] number of locally-supported subdomains = 2
>>>    Subdomain solver info is as follows:
>>>      - - - - - - - - - - - - - - - - - -
>>>      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>    KSP Object: (sub_) 1 MPI processes
>>>      type: fgmres
>>>        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>        happy breakdown tolerance 1e-30
>>>      maximum iterations=10000, initial guess is zero
>>>      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>      right preconditioning
>>>      using UNPRECONDITIONED norm type for convergence test
>>>    PC Object: (sub_) 1 MPI processes
>>>      type: ilu
>>>        out-of-place factorization
>>>        0 levels of fill
>>>        tolerance for zero pivot 2.22045e-14
>>>        matrix ordering: natural
>>>        factor fill ratio given 1., needed 1.
>>>          Factored matrix follows:
>>>            Mat Object: 1 MPI processes
>>>              type: seqaij
>>>              rows=484, cols=484
>>>              package used to perform factorization: petsc
>>>              total: nonzeros=15376, allocated nonzeros=15376
>>>              total number of mallocs used during MatSetValues calls =0
>>>                not using I-node routines
>>>      linear system matrix = precond matrix:
>>>      Mat Object: () 1 MPI processes
>>>        type: seqaij
>>>        rows=484, cols=484
>>>        total: nonzeros=15376, allocated nonzeros=15376
>>>        total number of mallocs used during MatSetValues calls =0
>>>          not using I-node routines
>>>      - - - - - - - - - - - - - - - - - -
>>>      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>    KSP Object: (sub_) 1 MPI processes
>>>      type: fgmres
>>>        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>        happy breakdown tolerance 1e-30
>>>      maximum iterations=10000, initial guess is zero
>>>      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>      right preconditioning
>>>      using UNPRECONDITIONED norm type for convergence test
>>>    PC Object: (sub_) 1 MPI processes
>>>      type: ilu
>>>        out-of-place factorization
>>>        0 levels of fill
>>>        tolerance for zero pivot 2.22045e-14
>>>        matrix ordering: natural
>>>        factor fill ratio given 1., needed 1.
>>>          Factored matrix follows:
>>>            Mat Object: 1 MPI processes
>>>              type: seqaij
>>>              rows=121, cols=121
>>>              package used to perform factorization: petsc
>>>              total: nonzeros=961, allocated nonzeros=961
>>>              total number of mallocs used during MatSetValues calls =0
>>>                not using I-node routines
>>>      linear system matrix = precond matrix:
>>>      Mat Object: () 1 MPI processes
>>>        type: seqaij
>>>        rows=121, cols=121
>>>        total: nonzeros=961, allocated nonzeros=961
>>>        total number of mallocs used during MatSetValues calls =0
>>>          not using I-node routines
>>>      - - - - - - - - - - - - - - - - - -
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
>>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>> 
>>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html).
>>>> 
>>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>>> 
>>>> Fande,
>>>> 
>>>> 
>>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>>> 
>>>> -- Boyce
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/dbd70247/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Apr 30 22:41:29 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 1 May 2019 03:41:29 +0000
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <7A10C5F8-7841-4D9F-A07B-0CCE85BC3DF6@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
	<82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>
	<7A10C5F8-7841-4D9F-A07B-0CCE85BC3DF6@gmail.com>
Message-ID: <E35B2A9E-C9EA-440E-BD4C-C42475078C1B@mcs.anl.gov>


  Preconditioned residual falling nicely while true residual gets stuck usually indicates 1) null space not properly handled 2) nonlinearity (unintentional) inside PC 3) very small pivots in factorization

  What happens if you use -ksp_type fgmres ? Same behavior? -ksp_pc_side right (with gmres)?

  Could also be using uninitialized data inside GAMS which would mimic the unintentional nonlinearity in the PC, run under valgrind.

   Barry


> On Apr 30, 2019, at 10:33 PM, Boyce Griffith <boyceg at gmail.com> wrote:
> 
> 
>> On Apr 30, 2019, at 6:22 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Boyce,
>> 
>>    I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why!
> 
> Thanks for checking! I don?t know why either. I often use FGMRES for outer KSPs, and probably used it for the inner ones without thinking. However, switching to preonly does not appear to help here:
> 
> ==========
> 
> Using GMRES as the outer KSP with ILU for the subdomain PC:
> 
> ./main -d 3 -n 10 -ksp_type gmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
> 
>   0 KSP preconditioned resid norm 7.956172961039e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.790924766864e-01 true resid norm 6.415624322140e+01 ||r(i)||/||b|| 1.222031831626e-01
>   2 KSP preconditioned resid norm 1.021248567184e-02 true resid norm 1.531971188768e+00 ||r(i)||/||b|| 2.918059823653e-03
>   3 KSP preconditioned resid norm 1.245290521533e-03 true resid norm 1.344228387002e-01 ||r(i)||/||b|| 2.560452101634e-04
>   4 KSP preconditioned resid norm 1.090553693795e-04 true resid norm 1.055832655005e-01 ||r(i)||/||b|| 2.011123233687e-04
>   5 KSP preconditioned resid norm 5.774209753120e-06 true resid norm 1.045883881261e-01 ||r(i)||/||b|| 1.992173062059e-04
>   6 KSP preconditioned resid norm 4.856074176119e-07 true resid norm 1.047048288204e-01 ||r(i)||/||b|| 1.994390994840e-04
>   7 KSP preconditioned resid norm 3.569834574103e-08 true resid norm 1.046981572164e-01 ||r(i)||/||b|| 1.994263915821e-04
>   8 KSP preconditioned resid norm 1.914112077836e-09 true resid norm 1.046980716404e-01 ||r(i)||/||b|| 1.994262285791e-04
>   9 KSP preconditioned resid norm 9.380275080687e-11 true resid norm 1.046980942278e-01 ||r(i)||/||b|| 1.994262716030e-04
>  10 KSP preconditioned resid norm 3.490998066884e-12 true resid norm 1.046980929565e-01 ||r(i)||/||b|| 1.994262691815e-04
>  11 KSP preconditioned resid norm 2.275544655754e-13 true resid norm 1.046980929905e-01 ||r(i)||/||b|| 1.994262692463e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=20, nonzero initial guess
>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gasm
>     Restriction/interpolation type: RESTRICT
>     requested amount of overlap = 0
>     total number of subdomains = 2
>     max number of local subdomains = 2
>     [0|1] number of locally-supported subdomains = 2
>     Subdomain solver info is as follows:
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=484, cols=484
>               package used to perform factorization: petsc
>               total: nonzeros=15376, allocated nonzeros=15376
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=484, cols=484
>         total: nonzeros=15376, allocated nonzeros=15376
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=121, cols=121
>               package used to perform factorization: petsc
>               total: nonzeros=961, allocated nonzeros=961
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=121, cols=121
>         total: nonzeros=961, allocated nonzeros=961
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
> 
> This shows convergence in the preconditioned norm but not in the true norm, and the solution appears nonphysical.
> 
> ==========
> 
> Using GMRES for the outer KSP with LU for the subdomain PC:
> 
> ./main -d 3 -n 10 -ksp_type gmres -sub_pc_type lu -ksp_monitor_true_residual -ksp_view
> 
>   0 KSP preconditioned resid norm 7.587523490344e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.452294080360e-15 true resid norm 1.026159563147e-01 ||r(i)||/||b|| 1.954602681716e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=5000, nonzero initial guess
>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gasm
>     Restriction/interpolation type: RESTRICT
>     requested amount of overlap = 0
>     total number of subdomains = 2
>     max number of local subdomains = 2
>     [0|1] number of locally-supported subdomains = 2
>     Subdomain solver info is as follows:
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: lu
>         out-of-place factorization
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: nd
>         factor fill ratio given 5., needed 2.69979
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=484, cols=484
>               package used to perform factorization: petsc
>               total: nonzeros=41512, allocated nonzeros=41512
>               total number of mallocs used during MatSetValues calls =0
>                 using I-node routines: found 194 nodes, limit used is 5
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=484, cols=484
>         total: nonzeros=15376, allocated nonzeros=15376
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: lu
>         out-of-place factorization
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: nd
>         factor fill ratio given 5., needed 2.57128
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=121, cols=121
>               package used to perform factorization: petsc
>               total: nonzeros=2471, allocated nonzeros=2471
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=121, cols=121
>         total: nonzeros=961, allocated nonzeros=961
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
> 
> 
> 
> ==========
> 
> Using FGMRES as the outer KSP:
> 
> ./main -d 3 -n 10 -ksp_type fgmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
> 
>   0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 9.908062627364e+00 true resid norm 9.908062627364e+00 ||r(i)||/||b|| 1.887262612713e-02
>   2 KSP unpreconditioned resid norm 1.198244246143e+00 true resid norm 1.198244246143e+00 ||r(i)||/||b|| 2.282385216660e-03
>   3 KSP unpreconditioned resid norm 1.011712741080e-01 true resid norm 1.011712741080e-01 ||r(i)||/||b|| 1.927084741847e-04
>   4 KSP unpreconditioned resid norm 9.890789240724e-02 true resid norm 9.890789240713e-02 ||r(i)||/||b|| 1.883972421881e-04
>   5 KSP unpreconditioned resid norm 9.790912262081e-02 true resid norm 9.790912262142e-02 ||r(i)||/||b|| 1.864948108590e-04
>   6 KSP unpreconditioned resid norm 9.758733533827e-02 true resid norm 9.758733534329e-02 ||r(i)||/||b|| 1.858818786218e-04
>   7 KSP unpreconditioned resid norm 9.754239526505e-02 true resid norm 9.754239528968e-02 ||r(i)||/||b|| 1.857962779487e-04
>   8 KSP unpreconditioned resid norm 9.732735871034e-02 true resid norm 9.732735925876e-02 ||r(i)||/||b|| 1.853866827767e-04
>   9 KSP unpreconditioned resid norm 9.681037606662e-02 true resid norm 9.681036910553e-02 ||r(i)||/||b|| 1.844019330592e-04
>  10 KSP unpreconditioned resid norm 9.672261014893e-02 true resid norm 9.672255386871e-02 ||r(i)||/||b|| 1.842346648258e-04
>  11 KSP unpreconditioned resid norm 9.636675313899e-02 true resid norm 9.636858548398e-02 ||r(i)||/||b|| 1.835604348338e-04
>  12 KSP unpreconditioned resid norm 9.626809760271e-02 true resid norm 9.624997274413e-02 ||r(i)||/||b|| 1.833345042985e-04
>  13 KSP unpreconditioned resid norm 9.538127889520e-02 true resid norm 9.646040937528e-02 ||r(i)||/||b|| 1.837353386505e-04
>  14 KSP unpreconditioned resid norm 7.698037210447e-02 true resid norm 1.004972547621e-01 ||r(i)||/||b|| 1.914246192480e-04
>  15 KSP unpreconditioned resid norm 6.963471194663e-02 true resid norm 1.021051124617e-01 ||r(i)||/||b|| 1.944872257707e-04
>  16 KSP unpreconditioned resid norm 6.288795988348e-02 true resid norm 1.048077408458e-01 ||r(i)||/||b|| 1.996351236971e-04
>  17 KSP unpreconditioned resid norm 5.576252453977e-02 true resid norm 1.071287580389e-01 ||r(i)||/||b|| 2.040561383159e-04
>  18 KSP unpreconditioned resid norm 5.096717927451e-02 true resid norm 1.092140137471e-01 ||r(i)||/||b|| 2.080280804443e-04
>  19 KSP unpreconditioned resid norm 4.698350635435e-02 true resid norm 1.104283980783e-01 ||r(i)||/||b|| 2.103412088852e-04
>  20 KSP unpreconditioned resid norm 4.387914680239e-02 true resid norm 1.116698560164e-01 ||r(i)||/||b|| 2.127059064451e-04
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=20, nonzero initial guess
>   tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gasm
>     Restriction/interpolation type: RESTRICT
>     requested amount of overlap = 0
>     total number of subdomains = 2
>     max number of local subdomains = 2
>     [0|1] number of locally-supported subdomains = 2
>     Subdomain solver info is as follows:
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=484, cols=484
>               package used to perform factorization: petsc
>               total: nonzeros=15376, allocated nonzeros=15376
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=484, cols=484
>         total: nonzeros=15376, allocated nonzeros=15376
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
>       [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>     KSP Object: (sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI processes
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: 1 MPI processes
>               type: seqaij
>               rows=121, cols=121
>               package used to perform factorization: petsc
>               total: nonzeros=961, allocated nonzeros=961
>               total number of mallocs used during MatSetValues calls =0
>                 not using I-node routines
>       linear system matrix = precond matrix:
>       Mat Object: () 1 MPI processes
>         type: seqaij
>         rows=121, cols=121
>         total: nonzeros=961, allocated nonzeros=961
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>       - - - - - - - - - - - - - - - - - -
> 
> ==========
> 
> Again, note that using the same subdomains, ASM appears to work fine.
> 
>> This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? 
>> 
>>> On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> 
>>> 
>>>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>> 
>>>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>>>> 
>>>> This is not what I am seeing in this output.
>>> 
>>> Whoops, I switched the outer KSP from GMRES to FGMRES.
>>> 
>>> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
>>> 
>>> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>> 
>>>  0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>  1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>>>  2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>>>  3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>>>  4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>>>  5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>>>  6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>>>  7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>>>  8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>>>  9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>>> 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>>> 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
>>> 
>>>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
>>> 
>>> Yes indeed, the solution appears to be no good with this preconditioner.
>>> 
>>> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
>>> 
>>>> That is, do you check that your model is not messed up, like a bad mesh.
>>> 
>>> This is a uniform HEX mesh automatically generated by libMesh.
>>> 
>>>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>>>> 
>>>>>> 
>>>>>> That PC is singular. 
>>>>> 
>>>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>>>> 
>>>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>>>> 
>>>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>>>> 
>>>> Here you go:
>>>> 
>>>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>> make: `main' is up to date.
>>>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>> 
>>>>  0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>  1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>>>  2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>>>  3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>>>  4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>>>  5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>>>  6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>>>  7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>>>  8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>>>  9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>>> 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>>>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>>>> KSP Object: 1 MPI processes
>>>>  type: fgmres
>>>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>    happy breakdown tolerance 1e-30
>>>>  maximum iterations=10, nonzero initial guess
>>>>  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>  right preconditioning
>>>>  using UNPRECONDITIONED norm type for convergence test
>>>> PC Object: 1 MPI processes
>>>>  type: gasm
>>>>    Restriction/interpolation type: RESTRICT
>>>>    requested amount of overlap = 0
>>>>    total number of subdomains = 2
>>>>    max number of local subdomains = 2
>>>>    [0|1] number of locally-supported subdomains = 2
>>>>    Subdomain solver info is as follows:
>>>>      - - - - - - - - - - - - - - - - - -
>>>>      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>>    KSP Object: (sub_) 1 MPI processes
>>>>      type: fgmres
>>>>        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>        happy breakdown tolerance 1e-30
>>>>      maximum iterations=10000, initial guess is zero
>>>>      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>      right preconditioning
>>>>      using UNPRECONDITIONED norm type for convergence test
>>>>    PC Object: (sub_) 1 MPI processes
>>>>      type: ilu
>>>>        out-of-place factorization
>>>>        0 levels of fill
>>>>        tolerance for zero pivot 2.22045e-14
>>>>        matrix ordering: natural
>>>>        factor fill ratio given 1., needed 1.
>>>>          Factored matrix follows:
>>>>            Mat Object: 1 MPI processes
>>>>              type: seqaij
>>>>              rows=484, cols=484
>>>>              package used to perform factorization: petsc
>>>>              total: nonzeros=15376, allocated nonzeros=15376
>>>>              total number of mallocs used during MatSetValues calls =0
>>>>                not using I-node routines
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: () 1 MPI processes
>>>>        type: seqaij
>>>>        rows=484, cols=484
>>>>        total: nonzeros=15376, allocated nonzeros=15376
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          not using I-node routines
>>>>      - - - - - - - - - - - - - - - - - -
>>>>      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>>    KSP Object: (sub_) 1 MPI processes
>>>>      type: fgmres
>>>>        restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>        happy breakdown tolerance 1e-30
>>>>      maximum iterations=10000, initial guess is zero
>>>>      tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>      right preconditioning
>>>>      using UNPRECONDITIONED norm type for convergence test
>>>>    PC Object: (sub_) 1 MPI processes
>>>>      type: ilu
>>>>        out-of-place factorization
>>>>        0 levels of fill
>>>>        tolerance for zero pivot 2.22045e-14
>>>>        matrix ordering: natural
>>>>        factor fill ratio given 1., needed 1.
>>>>          Factored matrix follows:
>>>>            Mat Object: 1 MPI processes
>>>>              type: seqaij
>>>>              rows=121, cols=121
>>>>              package used to perform factorization: petsc
>>>>              total: nonzeros=961, allocated nonzeros=961
>>>>              total number of mallocs used during MatSetValues calls =0
>>>>                not using I-node routines
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: () 1 MPI processes
>>>>        type: seqaij
>>>>        rows=121, cols=121
>>>>        total: nonzeros=961, allocated nonzeros=961
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          not using I-node routines
>>>>      - - - - - - - - - - - - - - - - - -
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>>>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
>>>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>>>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>>>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>>>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>>>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>>>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>>>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>> 
>>>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html).
>>>>> 
>>>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>>>> 
>>>>> Fande,
>>>>> 
>>>>> 
>>>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>>>> 
>>>>> -- Boyce
>>>> 
>>> 
>> 
> 


From boyceg at gmail.com  Tue Apr 30 23:06:43 2019
From: boyceg at gmail.com (Boyce Griffith)
Date: Wed, 1 May 2019 00:06:43 -0400
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <E35B2A9E-C9EA-440E-BD4C-C42475078C1B@mcs.anl.gov>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
	<82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>
	<7A10C5F8-7841-4D9F-A07B-0CCE85BC3DF6@gmail.com>
	<E35B2A9E-C9EA-440E-BD4C-C42475078C1B@mcs.anl.gov>
Message-ID: <2923369E-5F09-4824-A709-C73BBC48C4F9@gmail.com>


> On Apr 30, 2019, at 11:41 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Preconditioned residual falling nicely while true residual gets stuck usually indicates 1) null space not properly handled 2) nonlinearity (unintentional) inside PC 3) very small pivots in factorization

Sure.

Regarding (1), in the problem, one of the subdomains is just Laplace, whereas the other subdomain is a coupled parabolic/elliptic system. The elliptic equations are coupled at the subdomain interface. ILU and LU are both able to factor the submatrices without complaint and without special nullspace handling. I will add an extra Dirichlet boundary condition and see if it makes any difference.

Avoiding (2) is why I usually use FGMRES as the outer solver.

However, if I setup the ?same? preconditioner but ASM instead of GASM, everything appears to work. (E.g. solver converges quickly with reasonable overlaps and relatively good subdomain solves.) I am doing everything in serial ? shouldn?t ASM and GASM do (nearly) the same thing?

Also, FWIW, I can do the same problem just using FIELDSPLIT, and that seems to converge pretty well.

>  What happens if you use -ksp_type fgmres ? Same behavior?

With FGMRES, the residual stalls out (around 1e-4).

> -ksp_pc_side right (with gmres)?

With GMRES and right preconditioning, the residual stalls out (around 1e-2).

>  Could also be using uninitialized data inside GAMS which would mimic the unintentional nonlinearity in the PC, run under valgrind.

Well, I actually get a seg fault from the code when I run with valgrind, but I am not sure that is real. (I try to use mpich with valgrind, but right now I am running OpenMPI on MacOS right now ? I recently got an updated mpich from homebrew that was breaking everything!) I will try to check this out on a different system to see if I can get more information.

>   Barry
> 
> 
>> On Apr 30, 2019, at 10:33 PM, Boyce Griffith <boyceg at gmail.com> wrote:
>> 
>> 
>>> On Apr 30, 2019, at 6:22 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Boyce,
>>> 
>>>   I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why!
>> 
>> Thanks for checking! I don?t know why either. I often use FGMRES for outer KSPs, and probably used it for the inner ones without thinking. However, switching to preonly does not appear to help here:
>> 
>> ==========
>> 
>> Using GMRES as the outer KSP with ILU for the subdomain PC:
>> 
>> ./main -d 3 -n 10 -ksp_type gmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
>> 
>>  0 KSP preconditioned resid norm 7.956172961039e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.790924766864e-01 true resid norm 6.415624322140e+01 ||r(i)||/||b|| 1.222031831626e-01
>>  2 KSP preconditioned resid norm 1.021248567184e-02 true resid norm 1.531971188768e+00 ||r(i)||/||b|| 2.918059823653e-03
>>  3 KSP preconditioned resid norm 1.245290521533e-03 true resid norm 1.344228387002e-01 ||r(i)||/||b|| 2.560452101634e-04
>>  4 KSP preconditioned resid norm 1.090553693795e-04 true resid norm 1.055832655005e-01 ||r(i)||/||b|| 2.011123233687e-04
>>  5 KSP preconditioned resid norm 5.774209753120e-06 true resid norm 1.045883881261e-01 ||r(i)||/||b|| 1.992173062059e-04
>>  6 KSP preconditioned resid norm 4.856074176119e-07 true resid norm 1.047048288204e-01 ||r(i)||/||b|| 1.994390994840e-04
>>  7 KSP preconditioned resid norm 3.569834574103e-08 true resid norm 1.046981572164e-01 ||r(i)||/||b|| 1.994263915821e-04
>>  8 KSP preconditioned resid norm 1.914112077836e-09 true resid norm 1.046980716404e-01 ||r(i)||/||b|| 1.994262285791e-04
>>  9 KSP preconditioned resid norm 9.380275080687e-11 true resid norm 1.046980942278e-01 ||r(i)||/||b|| 1.994262716030e-04
>> 10 KSP preconditioned resid norm 3.490998066884e-12 true resid norm 1.046980929565e-01 ||r(i)||/||b|| 1.994262691815e-04
>> 11 KSP preconditioned resid norm 2.275544655754e-13 true resid norm 1.046980929905e-01 ||r(i)||/||b|| 1.994262692463e-04
>> KSP Object: 1 MPI processes
>>  type: gmres
>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    happy breakdown tolerance 1e-30
>>  maximum iterations=20, nonzero initial guess
>>  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>  type: gasm
>>    Restriction/interpolation type: RESTRICT
>>    requested amount of overlap = 0
>>    total number of subdomains = 2
>>    max number of local subdomains = 2
>>    [0|1] number of locally-supported subdomains = 2
>>    Subdomain solver info is as follows:
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: ilu
>>        out-of-place factorization
>>        0 levels of fill
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: natural
>>        factor fill ratio given 1., needed 1.
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=484, cols=484
>>              package used to perform factorization: petsc
>>              total: nonzeros=15376, allocated nonzeros=15376
>>              total number of mallocs used during MatSetValues calls =0
>>                not using I-node routines
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=484, cols=484
>>        total: nonzeros=15376, allocated nonzeros=15376
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: ilu
>>        out-of-place factorization
>>        0 levels of fill
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: natural
>>        factor fill ratio given 1., needed 1.
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=121, cols=121
>>              package used to perform factorization: petsc
>>              total: nonzeros=961, allocated nonzeros=961
>>              total number of mallocs used during MatSetValues calls =0
>>                not using I-node routines
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=121, cols=121
>>        total: nonzeros=961, allocated nonzeros=961
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>> 
>> This shows convergence in the preconditioned norm but not in the true norm, and the solution appears nonphysical.
>> 
>> ==========
>> 
>> Using GMRES for the outer KSP with LU for the subdomain PC:
>> 
>> ./main -d 3 -n 10 -ksp_type gmres -sub_pc_type lu -ksp_monitor_true_residual -ksp_view
>> 
>>  0 KSP preconditioned resid norm 7.587523490344e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP preconditioned resid norm 1.452294080360e-15 true resid norm 1.026159563147e-01 ||r(i)||/||b|| 1.954602681716e-04
>> KSP Object: 1 MPI processes
>>  type: gmres
>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    happy breakdown tolerance 1e-30
>>  maximum iterations=5000, nonzero initial guess
>>  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>  type: gasm
>>    Restriction/interpolation type: RESTRICT
>>    requested amount of overlap = 0
>>    total number of subdomains = 2
>>    max number of local subdomains = 2
>>    [0|1] number of locally-supported subdomains = 2
>>    Subdomain solver info is as follows:
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: lu
>>        out-of-place factorization
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: nd
>>        factor fill ratio given 5., needed 2.69979
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=484, cols=484
>>              package used to perform factorization: petsc
>>              total: nonzeros=41512, allocated nonzeros=41512
>>              total number of mallocs used during MatSetValues calls =0
>>                using I-node routines: found 194 nodes, limit used is 5
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=484, cols=484
>>        total: nonzeros=15376, allocated nonzeros=15376
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: lu
>>        out-of-place factorization
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: nd
>>        factor fill ratio given 5., needed 2.57128
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=121, cols=121
>>              package used to perform factorization: petsc
>>              total: nonzeros=2471, allocated nonzeros=2471
>>              total number of mallocs used during MatSetValues calls =0
>>                not using I-node routines
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=121, cols=121
>>        total: nonzeros=961, allocated nonzeros=961
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>> 
>> 
>> 
>> ==========
>> 
>> Using FGMRES as the outer KSP:
>> 
>> ./main -d 3 -n 10 -ksp_type fgmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
>> 
>>  0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>  1 KSP unpreconditioned resid norm 9.908062627364e+00 true resid norm 9.908062627364e+00 ||r(i)||/||b|| 1.887262612713e-02
>>  2 KSP unpreconditioned resid norm 1.198244246143e+00 true resid norm 1.198244246143e+00 ||r(i)||/||b|| 2.282385216660e-03
>>  3 KSP unpreconditioned resid norm 1.011712741080e-01 true resid norm 1.011712741080e-01 ||r(i)||/||b|| 1.927084741847e-04
>>  4 KSP unpreconditioned resid norm 9.890789240724e-02 true resid norm 9.890789240713e-02 ||r(i)||/||b|| 1.883972421881e-04
>>  5 KSP unpreconditioned resid norm 9.790912262081e-02 true resid norm 9.790912262142e-02 ||r(i)||/||b|| 1.864948108590e-04
>>  6 KSP unpreconditioned resid norm 9.758733533827e-02 true resid norm 9.758733534329e-02 ||r(i)||/||b|| 1.858818786218e-04
>>  7 KSP unpreconditioned resid norm 9.754239526505e-02 true resid norm 9.754239528968e-02 ||r(i)||/||b|| 1.857962779487e-04
>>  8 KSP unpreconditioned resid norm 9.732735871034e-02 true resid norm 9.732735925876e-02 ||r(i)||/||b|| 1.853866827767e-04
>>  9 KSP unpreconditioned resid norm 9.681037606662e-02 true resid norm 9.681036910553e-02 ||r(i)||/||b|| 1.844019330592e-04
>> 10 KSP unpreconditioned resid norm 9.672261014893e-02 true resid norm 9.672255386871e-02 ||r(i)||/||b|| 1.842346648258e-04
>> 11 KSP unpreconditioned resid norm 9.636675313899e-02 true resid norm 9.636858548398e-02 ||r(i)||/||b|| 1.835604348338e-04
>> 12 KSP unpreconditioned resid norm 9.626809760271e-02 true resid norm 9.624997274413e-02 ||r(i)||/||b|| 1.833345042985e-04
>> 13 KSP unpreconditioned resid norm 9.538127889520e-02 true resid norm 9.646040937528e-02 ||r(i)||/||b|| 1.837353386505e-04
>> 14 KSP unpreconditioned resid norm 7.698037210447e-02 true resid norm 1.004972547621e-01 ||r(i)||/||b|| 1.914246192480e-04
>> 15 KSP unpreconditioned resid norm 6.963471194663e-02 true resid norm 1.021051124617e-01 ||r(i)||/||b|| 1.944872257707e-04
>> 16 KSP unpreconditioned resid norm 6.288795988348e-02 true resid norm 1.048077408458e-01 ||r(i)||/||b|| 1.996351236971e-04
>> 17 KSP unpreconditioned resid norm 5.576252453977e-02 true resid norm 1.071287580389e-01 ||r(i)||/||b|| 2.040561383159e-04
>> 18 KSP unpreconditioned resid norm 5.096717927451e-02 true resid norm 1.092140137471e-01 ||r(i)||/||b|| 2.080280804443e-04
>> 19 KSP unpreconditioned resid norm 4.698350635435e-02 true resid norm 1.104283980783e-01 ||r(i)||/||b|| 2.103412088852e-04
>> 20 KSP unpreconditioned resid norm 4.387914680239e-02 true resid norm 1.116698560164e-01 ||r(i)||/||b|| 2.127059064451e-04
>> KSP Object: 1 MPI processes
>>  type: fgmres
>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    happy breakdown tolerance 1e-30
>>  maximum iterations=20, nonzero initial guess
>>  tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>  right preconditioning
>>  using UNPRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>  type: gasm
>>    Restriction/interpolation type: RESTRICT
>>    requested amount of overlap = 0
>>    total number of subdomains = 2
>>    max number of local subdomains = 2
>>    [0|1] number of locally-supported subdomains = 2
>>    Subdomain solver info is as follows:
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: ilu
>>        out-of-place factorization
>>        0 levels of fill
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: natural
>>        factor fill ratio given 1., needed 1.
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=484, cols=484
>>              package used to perform factorization: petsc
>>              total: nonzeros=15376, allocated nonzeros=15376
>>              total number of mallocs used during MatSetValues calls =0
>>                not using I-node routines
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=484, cols=484
>>        total: nonzeros=15376, allocated nonzeros=15376
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>>      [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>    KSP Object: (sub_) 1 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (sub_) 1 MPI processes
>>      type: ilu
>>        out-of-place factorization
>>        0 levels of fill
>>        tolerance for zero pivot 2.22045e-14
>>        matrix ordering: natural
>>        factor fill ratio given 1., needed 1.
>>          Factored matrix follows:
>>            Mat Object: 1 MPI processes
>>              type: seqaij
>>              rows=121, cols=121
>>              package used to perform factorization: petsc
>>              total: nonzeros=961, allocated nonzeros=961
>>              total number of mallocs used during MatSetValues calls =0
>>                not using I-node routines
>>      linear system matrix = precond matrix:
>>      Mat Object: () 1 MPI processes
>>        type: seqaij
>>        rows=121, cols=121
>>        total: nonzeros=961, allocated nonzeros=961
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node routines
>>      - - - - - - - - - - - - - - - - - -
>> 
>> ==========
>> 
>> Again, note that using the same subdomains, ASM appears to work fine.
>> 
>>> This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? 
>>> 
>>>> On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>> 
>>>>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>>>>> 
>>>>> This is not what I am seeing in this output.
>>>> 
>>>> Whoops, I switched the outer KSP from GMRES to FGMRES.
>>>> 
>>>> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
>>>> 
>>>> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>>> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>>> 
>>>> 0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>> 1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>>>> 2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>>>> 3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>>>> 4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>>>> 5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>>>> 6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>>>> 7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>>>> 8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>>>> 9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>>>> 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>>>> 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
>>>> 
>>>>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
>>>> 
>>>> Yes indeed, the solution appears to be no good with this preconditioner.
>>>> 
>>>> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
>>>> 
>>>>> That is, do you check that your model is not messed up, like a bad mesh.
>>>> 
>>>> This is a uniform HEX mesh automatically generated by libMesh.
>>>> 
>>>>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>>>>> 
>>>>>>> 
>>>>>>> That PC is singular. 
>>>>>> 
>>>>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>>>>> 
>>>>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>>>>> 
>>>>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>>>>> 
>>>>> Here you go:
>>>>> 
>>>>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>>> make: `main' is up to date.
>>>>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>>> 
>>>>> 0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>> 1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>>>> 2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>>>> 3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>>>> 4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>>>> 5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>>>> 6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>>>> 7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>>>> 8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>>>> 9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>>>> 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>>>>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>>>>> KSP Object: 1 MPI processes
>>>>> type: fgmres
>>>>>   restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>   happy breakdown tolerance 1e-30
>>>>> maximum iterations=10, nonzero initial guess
>>>>> tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>> right preconditioning
>>>>> using UNPRECONDITIONED norm type for convergence test
>>>>> PC Object: 1 MPI processes
>>>>> type: gasm
>>>>>   Restriction/interpolation type: RESTRICT
>>>>>   requested amount of overlap = 0
>>>>>   total number of subdomains = 2
>>>>>   max number of local subdomains = 2
>>>>>   [0|1] number of locally-supported subdomains = 2
>>>>>   Subdomain solver info is as follows:
>>>>>     - - - - - - - - - - - - - - - - - -
>>>>>     [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>>>   KSP Object: (sub_) 1 MPI processes
>>>>>     type: fgmres
>>>>>       restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>       happy breakdown tolerance 1e-30
>>>>>     maximum iterations=10000, initial guess is zero
>>>>>     tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>>     right preconditioning
>>>>>     using UNPRECONDITIONED norm type for convergence test
>>>>>   PC Object: (sub_) 1 MPI processes
>>>>>     type: ilu
>>>>>       out-of-place factorization
>>>>>       0 levels of fill
>>>>>       tolerance for zero pivot 2.22045e-14
>>>>>       matrix ordering: natural
>>>>>       factor fill ratio given 1., needed 1.
>>>>>         Factored matrix follows:
>>>>>           Mat Object: 1 MPI processes
>>>>>             type: seqaij
>>>>>             rows=484, cols=484
>>>>>             package used to perform factorization: petsc
>>>>>             total: nonzeros=15376, allocated nonzeros=15376
>>>>>             total number of mallocs used during MatSetValues calls =0
>>>>>               not using I-node routines
>>>>>     linear system matrix = precond matrix:
>>>>>     Mat Object: () 1 MPI processes
>>>>>       type: seqaij
>>>>>       rows=484, cols=484
>>>>>       total: nonzeros=15376, allocated nonzeros=15376
>>>>>       total number of mallocs used during MatSetValues calls =0
>>>>>         not using I-node routines
>>>>>     - - - - - - - - - - - - - - - - - -
>>>>>     [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>>>   KSP Object: (sub_) 1 MPI processes
>>>>>     type: fgmres
>>>>>       restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>       happy breakdown tolerance 1e-30
>>>>>     maximum iterations=10000, initial guess is zero
>>>>>     tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>>     right preconditioning
>>>>>     using UNPRECONDITIONED norm type for convergence test
>>>>>   PC Object: (sub_) 1 MPI processes
>>>>>     type: ilu
>>>>>       out-of-place factorization
>>>>>       0 levels of fill
>>>>>       tolerance for zero pivot 2.22045e-14
>>>>>       matrix ordering: natural
>>>>>       factor fill ratio given 1., needed 1.
>>>>>         Factored matrix follows:
>>>>>           Mat Object: 1 MPI processes
>>>>>             type: seqaij
>>>>>             rows=121, cols=121
>>>>>             package used to perform factorization: petsc
>>>>>             total: nonzeros=961, allocated nonzeros=961
>>>>>             total number of mallocs used during MatSetValues calls =0
>>>>>               not using I-node routines
>>>>>     linear system matrix = precond matrix:
>>>>>     Mat Object: () 1 MPI processes
>>>>>       type: seqaij
>>>>>       rows=121, cols=121
>>>>>       total: nonzeros=961, allocated nonzeros=961
>>>>>       total number of mallocs used during MatSetValues calls =0
>>>>>         not using I-node routines
>>>>>     - - - - - - - - - - - - - - - - - -
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>>>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>>>>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
>>>>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>>>>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>>>>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>>>>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>>>>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>>>>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>>>>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>>> 
>>>>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html).
>>>>>> 
>>>>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>>>>> 
>>>>>> Fande,
>>>>>> 
>>>>>> 
>>>>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>>>>> 
>>>>>> -- Boyce
>>>>> 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Tue Apr 30 23:49:53 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 1 May 2019 04:49:53 +0000
Subject: [petsc-users] ASM vs GASM
In-Reply-To: <2923369E-5F09-4824-A709-C73BBC48C4F9@gmail.com>
References: <7AE5F1B0-7753-48B2-B31B-42222A56C397@ad.unc.edu>
	<AF705A91-0192-4EC4-910B-31E86E5407D2@anl.gov>
	<A20419AD-920E-49B2-A507-8D9A52384B1A@gmail.com>
	<CE760A52-7330-45C5-B0FA-80346EB3B4E1@gmail.com>
	<CADOhEh6seGQ1RFeCHZv7L+Q9p8r6+zUBDcL1f2cYemS2U7Z7Xw@mail.gmail.com>
	<E98BA0FA-57F7-43D5-B8F0-6AEBD1AA3A8A@gmail.com>
	<CAN5Wd-LmBEp4E_efJXPcfkMvdoAykgcjBXZ9FK4rgsDVasHCLw@mail.gmail.com>
	<24FB1937-9B1E-4F79-951F-2E9BC96423CD@gmail.com>
	<CADOhEh6UAd6Xv5FcKz3WBGPLJQ7cj=3ptgLv71g-kWLD23Ke1A@mail.gmail.com>
	<1F4F719A-6C7B-4684-8DCB-FB82DA5532DC@gmail.com>
	<82DB3738-C9D1-45A6-8FF6-C09F70B26FDC@anl.gov>
	<7A10C5F8-7841-4D9F-A07B-0CCE85BC3DF6@gmail.com>
	<E35B2A9E-C9EA-440E-BD4C-C42475078C1B@mcs.anl.gov>
	<2923369E-5F09-4824-A709-C73BBC48C4F9@gmail.com>
Message-ID: <92D0DB9F-F3F5-4D0E-A5AE-0608CD3C0C75@mcs.anl.gov>

1) valgrind
2) confirm the IS are identical for ASM and GASM (this is easy on one process). Confirm the KSP,PC for subdomains are the same.
3) run with convergence monitoring for two subdomains (with ASM and GASM), turn on richardson on the inner solvers so it will plot the 
convergence history. Do they start with the same residual? If not shows they are being setup differently.
4) conclude it is probably a bug in GASM (or GASM's error checking). Send the code to petsc-main at mcs.anl.gov if possible 

  Barry


> On Apr 30, 2019, at 11:06 PM, Boyce Griffith <boyceg at gmail.com> wrote:
> 
> 
>> On Apr 30, 2019, at 11:41 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Preconditioned residual falling nicely while true residual gets stuck usually indicates 1) null space not properly handled 2) nonlinearity (unintentional) inside PC 3) very small pivots in factorization
> 
> Sure.
> 
> Regarding (1), in the problem, one of the subdomains is just Laplace, whereas the other subdomain is a coupled parabolic/elliptic system. The elliptic equations are coupled at the subdomain interface. ILU and LU are both able to factor the submatrices without complaint and without special nullspace handling. I will add an extra Dirichlet boundary condition and see if it makes any difference.
> 
> Avoiding (2) is why I usually use FGMRES as the outer solver.
> 
> However, if I setup the ?same? preconditioner but ASM instead of GASM, everything appears to work. (E.g. solver converges quickly with reasonable overlaps and relatively good subdomain solves.) I am doing everything in serial ? shouldn?t ASM and GASM do (nearly) the same thing?
> 
> Also, FWIW, I can do the same problem just using FIELDSPLIT, and that seems to converge pretty well.
> 
>> What happens if you use -ksp_type fgmres ? Same behavior?
> 
> With FGMRES, the residual stalls out (around 1e-4).
> 
>> -ksp_pc_side right (with gmres)?
> 
> With GMRES and right preconditioning, the residual stalls out (around 1e-2).
> 
>> Could also be using uninitialized data inside GAMS which would mimic the unintentional nonlinearity in the PC, run under valgrind.
> 
> Well, I actually get a seg fault from the code when I run with valgrind, but I am not sure that is real. (I try to use mpich with valgrind, but right now I am running OpenMPI on MacOS right now ? I recently got an updated mpich from homebrew that was breaking everything!) I will try to check this out on a different system to see if I can get more information.
> 
>>  Barry
>> 
>> 
>>> On Apr 30, 2019, at 10:33 PM, Boyce Griffith <boyceg at gmail.com> wrote:
>>> 
>>> 
>>>> On Apr 30, 2019, at 6:22 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Boyce,
>>>> 
>>>>  I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why!
>>> 
>>> Thanks for checking! I don?t know why either. I often use FGMRES for outer KSPs, and probably used it for the inner ones without thinking. However, switching to preonly does not appear to help here:
>>> 
>>> ==========
>>> 
>>> Using GMRES as the outer KSP with ILU for the subdomain PC:
>>> 
>>> ./main -d 3 -n 10 -ksp_type gmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
>>> 
>>> 0 KSP preconditioned resid norm 7.956172961039e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>> 1 KSP preconditioned resid norm 1.790924766864e-01 true resid norm 6.415624322140e+01 ||r(i)||/||b|| 1.222031831626e-01
>>> 2 KSP preconditioned resid norm 1.021248567184e-02 true resid norm 1.531971188768e+00 ||r(i)||/||b|| 2.918059823653e-03
>>> 3 KSP preconditioned resid norm 1.245290521533e-03 true resid norm 1.344228387002e-01 ||r(i)||/||b|| 2.560452101634e-04
>>> 4 KSP preconditioned resid norm 1.090553693795e-04 true resid norm 1.055832655005e-01 ||r(i)||/||b|| 2.011123233687e-04
>>> 5 KSP preconditioned resid norm 5.774209753120e-06 true resid norm 1.045883881261e-01 ||r(i)||/||b|| 1.992173062059e-04
>>> 6 KSP preconditioned resid norm 4.856074176119e-07 true resid norm 1.047048288204e-01 ||r(i)||/||b|| 1.994390994840e-04
>>> 7 KSP preconditioned resid norm 3.569834574103e-08 true resid norm 1.046981572164e-01 ||r(i)||/||b|| 1.994263915821e-04
>>> 8 KSP preconditioned resid norm 1.914112077836e-09 true resid norm 1.046980716404e-01 ||r(i)||/||b|| 1.994262285791e-04
>>> 9 KSP preconditioned resid norm 9.380275080687e-11 true resid norm 1.046980942278e-01 ||r(i)||/||b|| 1.994262716030e-04
>>> 10 KSP preconditioned resid norm 3.490998066884e-12 true resid norm 1.046980929565e-01 ||r(i)||/||b|| 1.994262691815e-04
>>> 11 KSP preconditioned resid norm 2.275544655754e-13 true resid norm 1.046980929905e-01 ||r(i)||/||b|| 1.994262692463e-04
>>> KSP Object: 1 MPI processes
>>> type: gmres
>>>   restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>   happy breakdown tolerance 1e-30
>>> maximum iterations=20, nonzero initial guess
>>> tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 1 MPI processes
>>> type: gasm
>>>   Restriction/interpolation type: RESTRICT
>>>   requested amount of overlap = 0
>>>   total number of subdomains = 2
>>>   max number of local subdomains = 2
>>>   [0|1] number of locally-supported subdomains = 2
>>>   Subdomain solver info is as follows:
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: ilu
>>>       out-of-place factorization
>>>       0 levels of fill
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: natural
>>>       factor fill ratio given 1., needed 1.
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=484, cols=484
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=15376, allocated nonzeros=15376
>>>             total number of mallocs used during MatSetValues calls =0
>>>               not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=484, cols=484
>>>       total: nonzeros=15376, allocated nonzeros=15376
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: ilu
>>>       out-of-place factorization
>>>       0 levels of fill
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: natural
>>>       factor fill ratio given 1., needed 1.
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=121, cols=121
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=961, allocated nonzeros=961
>>>             total number of mallocs used during MatSetValues calls =0
>>>               not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=121, cols=121
>>>       total: nonzeros=961, allocated nonzeros=961
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>> 
>>> This shows convergence in the preconditioned norm but not in the true norm, and the solution appears nonphysical.
>>> 
>>> ==========
>>> 
>>> Using GMRES for the outer KSP with LU for the subdomain PC:
>>> 
>>> ./main -d 3 -n 10 -ksp_type gmres -sub_pc_type lu -ksp_monitor_true_residual -ksp_view
>>> 
>>> 0 KSP preconditioned resid norm 7.587523490344e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>> 1 KSP preconditioned resid norm 1.452294080360e-15 true resid norm 1.026159563147e-01 ||r(i)||/||b|| 1.954602681716e-04
>>> KSP Object: 1 MPI processes
>>> type: gmres
>>>   restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>   happy breakdown tolerance 1e-30
>>> maximum iterations=5000, nonzero initial guess
>>> tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 1 MPI processes
>>> type: gasm
>>>   Restriction/interpolation type: RESTRICT
>>>   requested amount of overlap = 0
>>>   total number of subdomains = 2
>>>   max number of local subdomains = 2
>>>   [0|1] number of locally-supported subdomains = 2
>>>   Subdomain solver info is as follows:
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: lu
>>>       out-of-place factorization
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: nd
>>>       factor fill ratio given 5., needed 2.69979
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=484, cols=484
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=41512, allocated nonzeros=41512
>>>             total number of mallocs used during MatSetValues calls =0
>>>               using I-node routines: found 194 nodes, limit used is 5
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=484, cols=484
>>>       total: nonzeros=15376, allocated nonzeros=15376
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: lu
>>>       out-of-place factorization
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: nd
>>>       factor fill ratio given 5., needed 2.57128
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=121, cols=121
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=2471, allocated nonzeros=2471
>>>             total number of mallocs used during MatSetValues calls =0
>>>               not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=121, cols=121
>>>       total: nonzeros=961, allocated nonzeros=961
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>> 
>>> 
>>> 
>>> ==========
>>> 
>>> Using FGMRES as the outer KSP:
>>> 
>>> ./main -d 3 -n 10 -ksp_type fgmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view
>>> 
>>> 0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>> 1 KSP unpreconditioned resid norm 9.908062627364e+00 true resid norm 9.908062627364e+00 ||r(i)||/||b|| 1.887262612713e-02
>>> 2 KSP unpreconditioned resid norm 1.198244246143e+00 true resid norm 1.198244246143e+00 ||r(i)||/||b|| 2.282385216660e-03
>>> 3 KSP unpreconditioned resid norm 1.011712741080e-01 true resid norm 1.011712741080e-01 ||r(i)||/||b|| 1.927084741847e-04
>>> 4 KSP unpreconditioned resid norm 9.890789240724e-02 true resid norm 9.890789240713e-02 ||r(i)||/||b|| 1.883972421881e-04
>>> 5 KSP unpreconditioned resid norm 9.790912262081e-02 true resid norm 9.790912262142e-02 ||r(i)||/||b|| 1.864948108590e-04
>>> 6 KSP unpreconditioned resid norm 9.758733533827e-02 true resid norm 9.758733534329e-02 ||r(i)||/||b|| 1.858818786218e-04
>>> 7 KSP unpreconditioned resid norm 9.754239526505e-02 true resid norm 9.754239528968e-02 ||r(i)||/||b|| 1.857962779487e-04
>>> 8 KSP unpreconditioned resid norm 9.732735871034e-02 true resid norm 9.732735925876e-02 ||r(i)||/||b|| 1.853866827767e-04
>>> 9 KSP unpreconditioned resid norm 9.681037606662e-02 true resid norm 9.681036910553e-02 ||r(i)||/||b|| 1.844019330592e-04
>>> 10 KSP unpreconditioned resid norm 9.672261014893e-02 true resid norm 9.672255386871e-02 ||r(i)||/||b|| 1.842346648258e-04
>>> 11 KSP unpreconditioned resid norm 9.636675313899e-02 true resid norm 9.636858548398e-02 ||r(i)||/||b|| 1.835604348338e-04
>>> 12 KSP unpreconditioned resid norm 9.626809760271e-02 true resid norm 9.624997274413e-02 ||r(i)||/||b|| 1.833345042985e-04
>>> 13 KSP unpreconditioned resid norm 9.538127889520e-02 true resid norm 9.646040937528e-02 ||r(i)||/||b|| 1.837353386505e-04
>>> 14 KSP unpreconditioned resid norm 7.698037210447e-02 true resid norm 1.004972547621e-01 ||r(i)||/||b|| 1.914246192480e-04
>>> 15 KSP unpreconditioned resid norm 6.963471194663e-02 true resid norm 1.021051124617e-01 ||r(i)||/||b|| 1.944872257707e-04
>>> 16 KSP unpreconditioned resid norm 6.288795988348e-02 true resid norm 1.048077408458e-01 ||r(i)||/||b|| 1.996351236971e-04
>>> 17 KSP unpreconditioned resid norm 5.576252453977e-02 true resid norm 1.071287580389e-01 ||r(i)||/||b|| 2.040561383159e-04
>>> 18 KSP unpreconditioned resid norm 5.096717927451e-02 true resid norm 1.092140137471e-01 ||r(i)||/||b|| 2.080280804443e-04
>>> 19 KSP unpreconditioned resid norm 4.698350635435e-02 true resid norm 1.104283980783e-01 ||r(i)||/||b|| 2.103412088852e-04
>>> 20 KSP unpreconditioned resid norm 4.387914680239e-02 true resid norm 1.116698560164e-01 ||r(i)||/||b|| 2.127059064451e-04
>>> KSP Object: 1 MPI processes
>>> type: fgmres
>>>   restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>   happy breakdown tolerance 1e-30
>>> maximum iterations=20, nonzero initial guess
>>> tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>> right preconditioning
>>> using UNPRECONDITIONED norm type for convergence test
>>> PC Object: 1 MPI processes
>>> type: gasm
>>>   Restriction/interpolation type: RESTRICT
>>>   requested amount of overlap = 0
>>>   total number of subdomains = 2
>>>   max number of local subdomains = 2
>>>   [0|1] number of locally-supported subdomains = 2
>>>   Subdomain solver info is as follows:
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: ilu
>>>       out-of-place factorization
>>>       0 levels of fill
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: natural
>>>       factor fill ratio given 1., needed 1.
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=484, cols=484
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=15376, allocated nonzeros=15376
>>>             total number of mallocs used during MatSetValues calls =0
>>>               not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=484, cols=484
>>>       total: nonzeros=15376, allocated nonzeros=15376
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>>     [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>   KSP Object: (sub_) 1 MPI processes
>>>     type: preonly
>>>     maximum iterations=10000, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object: (sub_) 1 MPI processes
>>>     type: ilu
>>>       out-of-place factorization
>>>       0 levels of fill
>>>       tolerance for zero pivot 2.22045e-14
>>>       matrix ordering: natural
>>>       factor fill ratio given 1., needed 1.
>>>         Factored matrix follows:
>>>           Mat Object: 1 MPI processes
>>>             type: seqaij
>>>             rows=121, cols=121
>>>             package used to perform factorization: petsc
>>>             total: nonzeros=961, allocated nonzeros=961
>>>             total number of mallocs used during MatSetValues calls =0
>>>               not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Mat Object: () 1 MPI processes
>>>       type: seqaij
>>>       rows=121, cols=121
>>>       total: nonzeros=961, allocated nonzeros=961
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node routines
>>>     - - - - - - - - - - - - - - - - - -
>>> 
>>> ==========
>>> 
>>> Again, note that using the same subdomains, ASM appears to work fine.
>>> 
>>>> This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? 
>>>> 
>>>>> On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Apr 30, 2019, at 12:31 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>> 
>>>>>> When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."
>>>>>> 
>>>>>> This is not what I am seeing in this output.
>>>>> 
>>>>> Whoops, I switched the outer KSP from GMRES to FGMRES.
>>>>> 
>>>>> With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):
>>>>> 
>>>>> $ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>>>> Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual
>>>>> 
>>>>> 0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>> 1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01
>>>>> 2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03
>>>>> 3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04
>>>>> 4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04
>>>>> 5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04
>>>>> 6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04
>>>>> 7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04
>>>>> 8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04
>>>>> 9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04
>>>>> 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04
>>>>> 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04
>>>>> 
>>>>>> It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?
>>>>> 
>>>>> Yes indeed, the solution appears to be no good with this preconditioner.
>>>>> 
>>>>> Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.
>>>>> 
>>>>>> That is, do you check that your model is not messed up, like a bad mesh.
>>>>> 
>>>>> This is a uniform HEX mesh automatically generated by libMesh.
>>>>> 
>>>>>> On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Apr 30, 2019, at 11:23 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Apr 30, 2019, at 9:06 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> That PC is singular. 
>>>>>>> 
>>>>>>> Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.
>>>>>>> 
>>>>>>> We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores.  Do you mind to start from our example? Or setup an example for us to demonstrate  your issues? 
>>>>>>> 
>>>>>>> At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.
>>>>>> 
>>>>>> Here you go:
>>>>>> 
>>>>>> $ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>>>> make: `main' is up to date.
>>>>>> Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view
>>>>>> 
>>>>>> 0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>> 1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04
>>>>>> 2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04
>>>>>> 3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04
>>>>>> 4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04
>>>>>> 5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04
>>>>>> 6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04
>>>>>> 7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04
>>>>>> 8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04
>>>>>> 9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04
>>>>>> 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04
>>>>>> Linear solve did not converge due to DIVERGED_ITS iterations 10
>>>>>> KSP Object: 1 MPI processes
>>>>>> type: fgmres
>>>>>>  restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>  happy breakdown tolerance 1e-30
>>>>>> maximum iterations=10, nonzero initial guess
>>>>>> tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>>> right preconditioning
>>>>>> using UNPRECONDITIONED norm type for convergence test
>>>>>> PC Object: 1 MPI processes
>>>>>> type: gasm
>>>>>>  Restriction/interpolation type: RESTRICT
>>>>>>  requested amount of overlap = 0
>>>>>>  total number of subdomains = 2
>>>>>>  max number of local subdomains = 2
>>>>>>  [0|1] number of locally-supported subdomains = 2
>>>>>>  Subdomain solver info is as follows:
>>>>>>    - - - - - - - - - - - - - - - - - -
>>>>>>    [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484
>>>>>>  KSP Object: (sub_) 1 MPI processes
>>>>>>    type: fgmres
>>>>>>      restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>      happy breakdown tolerance 1e-30
>>>>>>    maximum iterations=10000, initial guess is zero
>>>>>>    tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>>>    right preconditioning
>>>>>>    using UNPRECONDITIONED norm type for convergence test
>>>>>>  PC Object: (sub_) 1 MPI processes
>>>>>>    type: ilu
>>>>>>      out-of-place factorization
>>>>>>      0 levels of fill
>>>>>>      tolerance for zero pivot 2.22045e-14
>>>>>>      matrix ordering: natural
>>>>>>      factor fill ratio given 1., needed 1.
>>>>>>        Factored matrix follows:
>>>>>>          Mat Object: 1 MPI processes
>>>>>>            type: seqaij
>>>>>>            rows=484, cols=484
>>>>>>            package used to perform factorization: petsc
>>>>>>            total: nonzeros=15376, allocated nonzeros=15376
>>>>>>            total number of mallocs used during MatSetValues calls =0
>>>>>>              not using I-node routines
>>>>>>    linear system matrix = precond matrix:
>>>>>>    Mat Object: () 1 MPI processes
>>>>>>      type: seqaij
>>>>>>      rows=484, cols=484
>>>>>>      total: nonzeros=15376, allocated nonzeros=15376
>>>>>>      total number of mallocs used during MatSetValues calls =0
>>>>>>        not using I-node routines
>>>>>>    - - - - - - - - - - - - - - - - - -
>>>>>>    [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121
>>>>>>  KSP Object: (sub_) 1 MPI processes
>>>>>>    type: fgmres
>>>>>>      restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>      happy breakdown tolerance 1e-30
>>>>>>    maximum iterations=10000, initial guess is zero
>>>>>>    tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>>>    right preconditioning
>>>>>>    using UNPRECONDITIONED norm type for convergence test
>>>>>>  PC Object: (sub_) 1 MPI processes
>>>>>>    type: ilu
>>>>>>      out-of-place factorization
>>>>>>      0 levels of fill
>>>>>>      tolerance for zero pivot 2.22045e-14
>>>>>>      matrix ordering: natural
>>>>>>      factor fill ratio given 1., needed 1.
>>>>>>        Factored matrix follows:
>>>>>>          Mat Object: 1 MPI processes
>>>>>>            type: seqaij
>>>>>>            rows=121, cols=121
>>>>>>            package used to perform factorization: petsc
>>>>>>            total: nonzeros=961, allocated nonzeros=961
>>>>>>            total number of mallocs used during MatSetValues calls =0
>>>>>>              not using I-node routines
>>>>>>    linear system matrix = precond matrix:
>>>>>>    Mat Object: () 1 MPI processes
>>>>>>      type: seqaij
>>>>>>      rows=121, cols=121
>>>>>>      total: nonzeros=961, allocated nonzeros=961
>>>>>>      total number of mallocs used during MatSetValues calls =0
>>>>>>        not using I-node routines
>>>>>>    - - - - - - - - - - - - - - - - - -
>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>>>>> [0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()
>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.11.1, unknown 
>>>>>> [0]PETSC ERROR: ./main on a darwin-dbg named boycesimacwork.dhcp.unc.edu by boyceg Tue Apr 30 11:29:02 2019
>>>>>> [0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local
>>>>>> [0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c
>>>>>> [0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c
>>>>>> [0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c
>>>>>> [0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c
>>>>>> [0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c
>>>>>> [0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c
>>>>>> 
>>>>>>> One difference between them is that the default GASM overlap is 0 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html), but the default ASM overlap is 1 (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html).
>>>>>>> 
>>>>>>> For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain.   It is a definitely interesting application for GASM. 
>>>>>>> 
>>>>>>> Fande,
>>>>>>> 
>>>>>>> 
>>>>>>> However, changing the GASM overlap does not make any difference in the convergence history.
>>>>>>> 
>>>>>>> -- Boyce
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>