From jychang48 at gmail.com  Tue Jan  1 04:06:16 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 1 Jan 2019 03:06:16 -0700
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
Message-ID: <CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>

Okay so I managed to successfully do this by translating the IS's into a
PetscSection, assigning it to an empty DMShell, and then assigning that to
KSP.

However, I seem to be unable to rename the outer fields that aggregate two
or more inner fields. Consider this snippet of a four-field dual
porosity/permeability FEniCS/petsc4py code:

...

## Extract FEniCS dof layout, global indices ##
dof_total = np.array(W.dofmap().dofs())
dof_v1 = np.array(W.sub(0).dofmap().dofs())
dof_p1 = np.array(W.sub(1).dofmap().dofs())
dof_v2 = np.array(W.sub(2).dofmap().dofs())
dof_p2 = np.array(W.sub(3).dofmap().dofs())
offset = np.min(dof_total)

## Create PetscSection ##
section = PETSc.Section().create()
section.setNumFields(4)
section.setFieldName(0,'v1')
section.setFieldName(1,'p1')
section.setFieldName(2,'v2')
section.setFieldName(3,'p2')
section.setFieldComponents(0,1)
section.setFieldComponents(1,1)
section.setFieldComponents(2,1)
section.setFieldComponents(3,1)
section.setChart(0,len(dof_total))
for i in np.nditer(dof_v1):
  section.setDof(i-offset,1)
  section.setFieldDof(i-offset,0,1)
for i in np.nditer(dof_p1):
  section.setDof(i-offset,1)
  section.setFieldDof(i-offset,1,1)
for i in np.nditer(dof_v2):
  section.setDof(i-offset,1)
  section.setFieldDof(i-offset,2,1)
for i in np.nditer(dof_p2):
  section.setDof(i-offset,1)
  section.setFieldDof(i-offset,3,1)
section.setUp()

## Create DM and assign PetscSection ##
dm = PETSc.DMShell().create()
dm.setDefaultSection(section)
dm.setUp()

## Create KSP and assign DM ##
ksp = PETSc.KSP().create()
ksp.setDM(dm)
ksp.setDMActive(False)

### PETSc Command-line options ##
PETScOptions.set('ksp_monitor_true_residual')
PETScOptions.set('ksp_view')
PETScOptions.set('ksp_type','gmres')
PETScOptions.set('pc_type','fieldsplit')
PETScOptions.set('pc_fieldsplit_0_fields','0,1')
PETScOptions.set('pc_fieldsplit_1_fields','2,3')
PETScOptions.set('pc_fieldsplit_type','additive')
PETScOptions.set('fieldsplit_0_ksp_type','preonly')
PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
PETScOptions.set('fieldsplit_1_ksp_type','preonly')
PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')

...

Is it possible to rename the outer splits (aka make
'-pc_fieldsplit_0_fields 0,1' be something like
'-pc_fieldsplit_micro_fields 0,1')?

Thanks,
Justin


On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi all,
>>
>> I am solving a six field battery problem (concentration and potential for
>> each of the two solid and one electrolyte domains) and I want to experiment
>> with nested/recursice fieldsplitting. I have the IS's and am able to use
>> these to define my PCFieldSplitsSetIS(). However, I can imagine this
>> getting really messy from a programming standpoint, especially once I need
>> to add temperature into the mix, so it is my hope that I can translate
>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>> rely on command line options to play around with various combinations of
>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>
>> However, it doesn't seem clear to me how I would create a DM when you
>> already know the IS's for each fields. If I understand functions like
>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>> IS's and sub DM's associated with the original DM, whereas I want to do it
>> the other way around. Perhaps the "reversal" of something like
>> DMCreateFieldIS()
>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>> where you convert the IS into a PetscSection or is there an easier/better
>> way?
>>
>> Any thoughts/help appreciated!
>>
>
> Paul has recently done this for LibMesh. I believe that constructing a
> PetscSection is enough to get you minimally started. That allows
> DMCreateSubDM() to work by subsetting the Section, and that should allow
> the command line to work. CreateFieldDecomposition() should
> be removed I think.
>
>   Thanks,
>
>     Matt
>
>
>> Justin
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190101/ead87370/attachment.html>

From knepley at gmail.com  Tue Jan  1 08:10:13 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Jan 2019 09:10:13 -0500
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
Message-ID: <CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>

On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com> wrote:

> Okay so I managed to successfully do this by translating the IS's into a
> PetscSection, assigning it to an empty DMShell, and then assigning that to
> KSP.
>
> However, I seem to be unable to rename the outer fields that aggregate two
> or more inner fields.
>

Right now you cannot do that because of the way that I check for the option:

https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369

I guess we could replace this by some search code that looked for any
option of that form and then read
out the splitname using scanf. Right now, I do not see a pattern search
function for the options database.

  Thanks,

    Matt


> Consider this snippet of a four-field dual porosity/permeability
> FEniCS/petsc4py code:
>
> ...
>
> ## Extract FEniCS dof layout, global indices ##
> dof_total = np.array(W.dofmap().dofs())
> dof_v1 = np.array(W.sub(0).dofmap().dofs())
> dof_p1 = np.array(W.sub(1).dofmap().dofs())
> dof_v2 = np.array(W.sub(2).dofmap().dofs())
> dof_p2 = np.array(W.sub(3).dofmap().dofs())
> offset = np.min(dof_total)
>
> ## Create PetscSection ##
> section = PETSc.Section().create()
> section.setNumFields(4)
> section.setFieldName(0,'v1')
> section.setFieldName(1,'p1')
> section.setFieldName(2,'v2')
> section.setFieldName(3,'p2')
> section.setFieldComponents(0,1)
> section.setFieldComponents(1,1)
> section.setFieldComponents(2,1)
> section.setFieldComponents(3,1)
> section.setChart(0,len(dof_total))
> for i in np.nditer(dof_v1):
>   section.setDof(i-offset,1)
>   section.setFieldDof(i-offset,0,1)
> for i in np.nditer(dof_p1):
>   section.setDof(i-offset,1)
>   section.setFieldDof(i-offset,1,1)
> for i in np.nditer(dof_v2):
>   section.setDof(i-offset,1)
>   section.setFieldDof(i-offset,2,1)
> for i in np.nditer(dof_p2):
>   section.setDof(i-offset,1)
>   section.setFieldDof(i-offset,3,1)
> section.setUp()
>
> ## Create DM and assign PetscSection ##
> dm = PETSc.DMShell().create()
> dm.setDefaultSection(section)
> dm.setUp()
>
> ## Create KSP and assign DM ##
> ksp = PETSc.KSP().create()
> ksp.setDM(dm)
> ksp.setDMActive(False)
>
> ### PETSc Command-line options ##
> PETScOptions.set('ksp_monitor_true_residual')
> PETScOptions.set('ksp_view')
> PETScOptions.set('ksp_type','gmres')
> PETScOptions.set('pc_type','fieldsplit')
> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
> PETScOptions.set('pc_fieldsplit_type','additive')
> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>
> ...
>
> Is it possible to rename the outer splits (aka make
> '-pc_fieldsplit_0_fields 0,1' be something like
> '-pc_fieldsplit_micro_fields 0,1')?
>
> Thanks,
> Justin
>
>
> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi all,
>>>
>>> I am solving a six field battery problem (concentration and potential
>>> for each of the two solid and one electrolyte domains) and I want to
>>> experiment with nested/recursice fieldsplitting. I have the IS's and am
>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>> this getting really messy from a programming standpoint, especially once I
>>> need to add temperature into the mix, so it is my hope that I can translate
>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>> rely on command line options to play around with various combinations of
>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>
>>> However, it doesn't seem clear to me how I would create a DM when you
>>> already know the IS's for each fields. If I understand functions like
>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>> the other way around. Perhaps the "reversal" of something like
>>> DMCreateFieldIS()
>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>> where you convert the IS into a PetscSection or is there an easier/better
>>> way?
>>>
>>> Any thoughts/help appreciated!
>>>
>>
>> Paul has recently done this for LibMesh. I believe that constructing a
>> PetscSection is enough to get you minimally started. That allows
>> DMCreateSubDM() to work by subsetting the Section, and that should allow
>> the command line to work. CreateFieldDecomposition() should
>> be removed I think.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Justin
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190101/145bb670/attachment-0001.html>

From jychang48 at gmail.com  Wed Jan  2 15:54:35 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 2 Jan 2019 14:54:35 -0700
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
	<CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
Message-ID: <CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>

Okay that's fine.

Now consider one more case:

Suppose I create a PetscSection with 9 fields:

section = PETSc.Section().create()
section.setNumFields(9)
section.setFieldName(0,'u1')
section.setFieldName(1,'c1')
section.setFieldName(2,'t1')
section.setFieldName(3,'u2')
section.setFieldName(4,'c2')
section.setFieldName(5,'t2')
section.setFieldName(6,'u3')
section.setFieldName(7,'c3')
section.setFieldName(8,'t3')
section.setFieldComponents(0,1)
section.setFieldComponents(1,1)
section.setFieldComponents(2,1)
section.setFieldComponents(3,1)
section.setFieldComponents(4,1)
section.setFieldComponents(5,1)
section.setFieldComponents(6,1)
section.setFieldComponents(7,1)
section.setFieldComponents(8,1)

....

where u is potential, c is concentration, and t is temperature. The numbers
refer to different domains of a battery (anode, electrolyte, cathode).

I want three levels of splits:
1) Split by domain
2) Split potential and concentration from temperature.
3) Split potential and concentration for schur complementing purpose.

Do the following PETSc command line options describe the split I mentioned
above:

-pc_type fieldsplit
-pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3,4,5
-pc_fieldsplit_2_fields 6,7,8
-fieldsplit_0_pc_type fieldsplit
-fieldsplit_1_pc_type fieldsplit
-fieldsplit_2_pc_type fieldsplit
-fieldsplit_0_pc_fieldsplit_0_fields 0,1
-fieldsplit_0_pc_fieldsplit_1_fields 2
-fieldsplit_1_pc_fieldsplit_0_fields 3,4
-fieldsplit_1_pc_fieldsplit_1_fields 5
-fieldsplit_2_pc_fieldsplit_0_fields 6,7
-fieldsplit_2_pc_fieldsplit_1_fields 8

I don't have an actual code to test these out with yet as I've never done
anything beyond two levels of split. So I am wondering if the above will
break anything as far as you can tell.

Thanks,
Justin

On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com> wrote:
>
>> Okay so I managed to successfully do this by translating the IS's into a
>> PetscSection, assigning it to an empty DMShell, and then assigning that to
>> KSP.
>>
>> However, I seem to be unable to rename the outer fields that aggregate
>> two or more inner fields.
>>
>
> Right now you cannot do that because of the way that I check for the
> option:
>
>
> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>
> I guess we could replace this by some search code that looked for any
> option of that form and then read
> out the splitname using scanf. Right now, I do not see a pattern search
> function for the options database.
>
>   Thanks,
>
>     Matt
>
>
>> Consider this snippet of a four-field dual porosity/permeability
>> FEniCS/petsc4py code:
>>
>> ...
>>
>> ## Extract FEniCS dof layout, global indices ##
>> dof_total = np.array(W.dofmap().dofs())
>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>> offset = np.min(dof_total)
>>
>> ## Create PetscSection ##
>> section = PETSc.Section().create()
>> section.setNumFields(4)
>> section.setFieldName(0,'v1')
>> section.setFieldName(1,'p1')
>> section.setFieldName(2,'v2')
>> section.setFieldName(3,'p2')
>> section.setFieldComponents(0,1)
>> section.setFieldComponents(1,1)
>> section.setFieldComponents(2,1)
>> section.setFieldComponents(3,1)
>> section.setChart(0,len(dof_total))
>> for i in np.nditer(dof_v1):
>>   section.setDof(i-offset,1)
>>   section.setFieldDof(i-offset,0,1)
>> for i in np.nditer(dof_p1):
>>   section.setDof(i-offset,1)
>>   section.setFieldDof(i-offset,1,1)
>> for i in np.nditer(dof_v2):
>>   section.setDof(i-offset,1)
>>   section.setFieldDof(i-offset,2,1)
>> for i in np.nditer(dof_p2):
>>   section.setDof(i-offset,1)
>>   section.setFieldDof(i-offset,3,1)
>> section.setUp()
>>
>> ## Create DM and assign PetscSection ##
>> dm = PETSc.DMShell().create()
>> dm.setDefaultSection(section)
>> dm.setUp()
>>
>> ## Create KSP and assign DM ##
>> ksp = PETSc.KSP().create()
>> ksp.setDM(dm)
>> ksp.setDMActive(False)
>>
>> ### PETSc Command-line options ##
>> PETScOptions.set('ksp_monitor_true_residual')
>> PETScOptions.set('ksp_view')
>> PETScOptions.set('ksp_type','gmres')
>> PETScOptions.set('pc_type','fieldsplit')
>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>> PETScOptions.set('pc_fieldsplit_type','additive')
>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>
>> ...
>>
>> Is it possible to rename the outer splits (aka make
>> '-pc_fieldsplit_0_fields 0,1' be something like
>> '-pc_fieldsplit_micro_fields 0,1')?
>>
>> Thanks,
>> Justin
>>
>>
>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am solving a six field battery problem (concentration and potential
>>>> for each of the two solid and one electrolyte domains) and I want to
>>>> experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>> this getting really messy from a programming standpoint, especially once I
>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>> rely on command line options to play around with various combinations of
>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>
>>>> However, it doesn't seem clear to me how I would create a DM when you
>>>> already know the IS's for each fields. If I understand functions like
>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>> the other way around. Perhaps the "reversal" of something like
>>>> DMCreateFieldIS()
>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>> way?
>>>>
>>>> Any thoughts/help appreciated!
>>>>
>>>
>>> Paul has recently done this for LibMesh. I believe that constructing a
>>> PetscSection is enough to get you minimally started. That allows
>>> DMCreateSubDM() to work by subsetting the Section, and that should allow
>>> the command line to work. CreateFieldDecomposition() should
>>> be removed I think.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Justin
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/802a5c77/attachment.html>

From jychang48 at gmail.com  Wed Jan  2 16:02:38 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 2 Jan 2019 15:02:38 -0700
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
	<CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
	<CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
Message-ID: <CAP2=TMgSzo2QyNyA+UXErEHrLRy++tiNQEGcVWzOUfWpw2-zXQ@mail.gmail.com>

Sorry, I forgot three more lines describing the 3rd level of split:

-pc_type fieldsplit
-pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3,4,5
-pc_fieldsplit_2_fields 6,7,8
-fieldsplit_0_pc_type fieldsplit
-fieldsplit_1_pc_type fieldsplit
-fieldsplit_2_pc_type fieldsplit
-fieldsplit_0_pc_fieldsplit_0_fields 0,1
-fieldsplit_0_pc_fieldsplit_1_fields 2
-fieldsplit_1_pc_fieldsplit_0_fields 3,4
-fieldsplit_1_pc_fieldsplit_1_fields 5
-fieldsplit_2_pc_fieldsplit_0_fields 6,7
-fieldsplit_2_pc_fieldsplit_1_fields 8

-fieldsplit_0_fieldsplit_0_pc_type fieldsplit
-fieldsplit_1_fieldsplit_0_pc_type fieldsplit
-fieldsplit_2_fieldsplit_0_pc_type fieldsplit

On Wed, Jan 2, 2019 at 2:54 PM Justin Chang <jychang48 at gmail.com> wrote:

> Okay that's fine.
>
> Now consider one more case:
>
> Suppose I create a PetscSection with 9 fields:
>
> section = PETSc.Section().create()
> section.setNumFields(9)
> section.setFieldName(0,'u1')
> section.setFieldName(1,'c1')
> section.setFieldName(2,'t1')
> section.setFieldName(3,'u2')
> section.setFieldName(4,'c2')
> section.setFieldName(5,'t2')
> section.setFieldName(6,'u3')
> section.setFieldName(7,'c3')
> section.setFieldName(8,'t3')
> section.setFieldComponents(0,1)
> section.setFieldComponents(1,1)
> section.setFieldComponents(2,1)
> section.setFieldComponents(3,1)
> section.setFieldComponents(4,1)
> section.setFieldComponents(5,1)
> section.setFieldComponents(6,1)
> section.setFieldComponents(7,1)
> section.setFieldComponents(8,1)
>
> ....
>
> where u is potential, c is concentration, and t is temperature. The
> numbers refer to different domains of a battery (anode, electrolyte,
> cathode).
>
> I want three levels of splits:
> 1) Split by domain
> 2) Split potential and concentration from temperature.
> 3) Split potential and concentration for schur complementing purpose.
>
> Do the following PETSc command line options describe the split I mentioned
> above:
>
> -pc_type fieldsplit
> -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3,4,5
> -pc_fieldsplit_2_fields 6,7,8
> -fieldsplit_0_pc_type fieldsplit
> -fieldsplit_1_pc_type fieldsplit
> -fieldsplit_2_pc_type fieldsplit
> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
> -fieldsplit_0_pc_fieldsplit_1_fields 2
> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
> -fieldsplit_1_pc_fieldsplit_1_fields 5
> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
> -fieldsplit_2_pc_fieldsplit_1_fields 8
>
> I don't have an actual code to test these out with yet as I've never done
> anything beyond two levels of split. So I am wondering if the above will
> break anything as far as you can tell.
>
> Thanks,
> Justin
>
> On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Okay so I managed to successfully do this by translating the IS's into a
>>> PetscSection, assigning it to an empty DMShell, and then assigning that to
>>> KSP.
>>>
>>> However, I seem to be unable to rename the outer fields that aggregate
>>> two or more inner fields.
>>>
>>
>> Right now you cannot do that because of the way that I check for the
>> option:
>>
>>
>> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>>
>> I guess we could replace this by some search code that looked for any
>> option of that form and then read
>> out the splitname using scanf. Right now, I do not see a pattern search
>> function for the options database.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Consider this snippet of a four-field dual porosity/permeability
>>> FEniCS/petsc4py code:
>>>
>>> ...
>>>
>>> ## Extract FEniCS dof layout, global indices ##
>>> dof_total = np.array(W.dofmap().dofs())
>>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>>> offset = np.min(dof_total)
>>>
>>> ## Create PetscSection ##
>>> section = PETSc.Section().create()
>>> section.setNumFields(4)
>>> section.setFieldName(0,'v1')
>>> section.setFieldName(1,'p1')
>>> section.setFieldName(2,'v2')
>>> section.setFieldName(3,'p2')
>>> section.setFieldComponents(0,1)
>>> section.setFieldComponents(1,1)
>>> section.setFieldComponents(2,1)
>>> section.setFieldComponents(3,1)
>>> section.setChart(0,len(dof_total))
>>> for i in np.nditer(dof_v1):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,0,1)
>>> for i in np.nditer(dof_p1):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,1,1)
>>> for i in np.nditer(dof_v2):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,2,1)
>>> for i in np.nditer(dof_p2):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,3,1)
>>> section.setUp()
>>>
>>> ## Create DM and assign PetscSection ##
>>> dm = PETSc.DMShell().create()
>>> dm.setDefaultSection(section)
>>> dm.setUp()
>>>
>>> ## Create KSP and assign DM ##
>>> ksp = PETSc.KSP().create()
>>> ksp.setDM(dm)
>>> ksp.setDMActive(False)
>>>
>>> ### PETSc Command-line options ##
>>> PETScOptions.set('ksp_monitor_true_residual')
>>> PETScOptions.set('ksp_view')
>>> PETScOptions.set('ksp_type','gmres')
>>> PETScOptions.set('pc_type','fieldsplit')
>>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>>> PETScOptions.set('pc_fieldsplit_type','additive')
>>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>>
>>> ...
>>>
>>> Is it possible to rename the outer splits (aka make
>>> '-pc_fieldsplit_0_fields 0,1' be something like
>>> '-pc_fieldsplit_micro_fields 0,1')?
>>>
>>> Thanks,
>>> Justin
>>>
>>>
>>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am solving a six field battery problem (concentration and potential
>>>>> for each of the two solid and one electrolyte domains) and I want to
>>>>> experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>>> this getting really messy from a programming standpoint, especially once I
>>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>>> rely on command line options to play around with various combinations of
>>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>>
>>>>> However, it doesn't seem clear to me how I would create a DM when you
>>>>> already know the IS's for each fields. If I understand functions like
>>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>>> the other way around. Perhaps the "reversal" of something like
>>>>> DMCreateFieldIS()
>>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>>> way?
>>>>>
>>>>> Any thoughts/help appreciated!
>>>>>
>>>>
>>>> Paul has recently done this for LibMesh. I believe that constructing a
>>>> PetscSection is enough to get you minimally started. That allows
>>>> DMCreateSubDM() to work by subsetting the Section, and that should
>>>> allow the command line to work. CreateFieldDecomposition() should
>>>> be removed I think.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Justin
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/76796313/attachment-0001.html>

From knepley at gmail.com  Wed Jan  2 16:38:54 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 Jan 2019 17:38:54 -0500
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
	<CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
	<CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
Message-ID: <CAMYG4GkDqg7LRG_pzVQ8dcofxw3pJRsT7urMjAtSsrjXenTPGg@mail.gmail.com>

On Wed, Jan 2, 2019 at 4:55 PM Justin Chang <jychang48 at gmail.com> wrote:

> Okay that's fine.
>
> Now consider one more case:
>
> Suppose I create a PetscSection with 9 fields:
>
> section = PETSc.Section().create()
> section.setNumFields(9)
> section.setFieldName(0,'u1')
> section.setFieldName(1,'c1')
> section.setFieldName(2,'t1')
> section.setFieldName(3,'u2')
> section.setFieldName(4,'c2')
> section.setFieldName(5,'t2')
> section.setFieldName(6,'u3')
> section.setFieldName(7,'c3')
> section.setFieldName(8,'t3')
> section.setFieldComponents(0,1)
> section.setFieldComponents(1,1)
> section.setFieldComponents(2,1)
> section.setFieldComponents(3,1)
> section.setFieldComponents(4,1)
> section.setFieldComponents(5,1)
> section.setFieldComponents(6,1)
> section.setFieldComponents(7,1)
> section.setFieldComponents(8,1)
>
> ....
>
> where u is potential, c is concentration, and t is temperature. The
> numbers refer to different domains of a battery (anode, electrolyte,
> cathode).
>
> I want three levels of splits:
> 1) Split by domain
> 2) Split potential and concentration from temperature.
> 3) Split potential and concentration for schur complementing purpose.
>
> Do the following PETSc command line options describe the split I mentioned
> above:
>
> -pc_type fieldsplit
> -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3,4,5
> -pc_fieldsplit_2_fields 6,7,8
> -fieldsplit_0_pc_type fieldsplit
> -fieldsplit_1_pc_type fieldsplit
> -fieldsplit_2_pc_type fieldsplit
> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
> -fieldsplit_0_pc_fieldsplit_1_fields 2
> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
> -fieldsplit_1_pc_fieldsplit_1_fields 5
> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
> -fieldsplit_2_pc_fieldsplit_1_fields 8
>
> I don't have an actual code to test these out with yet as I've never done
> anything beyond two levels of split. So I am wondering if the above will
> break anything as far as you can tell.
>

This should work as you expect. Send me the -ksp_view output when you run
it. It would be very cool to see 3-level in action.

  Thanks,

     Matt


> Thanks,
> Justin
>
> On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Okay so I managed to successfully do this by translating the IS's into a
>>> PetscSection, assigning it to an empty DMShell, and then assigning that to
>>> KSP.
>>>
>>> However, I seem to be unable to rename the outer fields that aggregate
>>> two or more inner fields.
>>>
>>
>> Right now you cannot do that because of the way that I check for the
>> option:
>>
>>
>> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>>
>> I guess we could replace this by some search code that looked for any
>> option of that form and then read
>> out the splitname using scanf. Right now, I do not see a pattern search
>> function for the options database.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Consider this snippet of a four-field dual porosity/permeability
>>> FEniCS/petsc4py code:
>>>
>>> ...
>>>
>>> ## Extract FEniCS dof layout, global indices ##
>>> dof_total = np.array(W.dofmap().dofs())
>>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>>> offset = np.min(dof_total)
>>>
>>> ## Create PetscSection ##
>>> section = PETSc.Section().create()
>>> section.setNumFields(4)
>>> section.setFieldName(0,'v1')
>>> section.setFieldName(1,'p1')
>>> section.setFieldName(2,'v2')
>>> section.setFieldName(3,'p2')
>>> section.setFieldComponents(0,1)
>>> section.setFieldComponents(1,1)
>>> section.setFieldComponents(2,1)
>>> section.setFieldComponents(3,1)
>>> section.setChart(0,len(dof_total))
>>> for i in np.nditer(dof_v1):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,0,1)
>>> for i in np.nditer(dof_p1):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,1,1)
>>> for i in np.nditer(dof_v2):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,2,1)
>>> for i in np.nditer(dof_p2):
>>>   section.setDof(i-offset,1)
>>>   section.setFieldDof(i-offset,3,1)
>>> section.setUp()
>>>
>>> ## Create DM and assign PetscSection ##
>>> dm = PETSc.DMShell().create()
>>> dm.setDefaultSection(section)
>>> dm.setUp()
>>>
>>> ## Create KSP and assign DM ##
>>> ksp = PETSc.KSP().create()
>>> ksp.setDM(dm)
>>> ksp.setDMActive(False)
>>>
>>> ### PETSc Command-line options ##
>>> PETScOptions.set('ksp_monitor_true_residual')
>>> PETScOptions.set('ksp_view')
>>> PETScOptions.set('ksp_type','gmres')
>>> PETScOptions.set('pc_type','fieldsplit')
>>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>>> PETScOptions.set('pc_fieldsplit_type','additive')
>>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>>
>>> ...
>>>
>>> Is it possible to rename the outer splits (aka make
>>> '-pc_fieldsplit_0_fields 0,1' be something like
>>> '-pc_fieldsplit_micro_fields 0,1')?
>>>
>>> Thanks,
>>> Justin
>>>
>>>
>>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am solving a six field battery problem (concentration and potential
>>>>> for each of the two solid and one electrolyte domains) and I want to
>>>>> experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>>> this getting really messy from a programming standpoint, especially once I
>>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>>> rely on command line options to play around with various combinations of
>>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>>
>>>>> However, it doesn't seem clear to me how I would create a DM when you
>>>>> already know the IS's for each fields. If I understand functions like
>>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>>> the other way around. Perhaps the "reversal" of something like
>>>>> DMCreateFieldIS()
>>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>>> way?
>>>>>
>>>>> Any thoughts/help appreciated!
>>>>>
>>>>
>>>> Paul has recently done this for LibMesh. I believe that constructing a
>>>> PetscSection is enough to get you minimally started. That allows
>>>> DMCreateSubDM() to work by subsetting the Section, and that should
>>>> allow the command line to work. CreateFieldDecomposition() should
>>>> be removed I think.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Justin
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/bbe646fa/attachment.html>

From fuentesdt at gmail.com  Wed Jan  2 18:27:05 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Wed, 2 Jan 2019 18:27:05 -0600
Subject: [petsc-users] DMPlexSetRefinementFunction
Message-ID: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>

Hi,

Starting with TS example 45

$ pwd
/opt/apps/PETSc/petsc-3.10.2
$ ls src/ts/examples/tutorials/ex45.c
src/ts/examples/tutorials/ex45.c

petsc configured with: ./config/configure.py --with-shared-libraries
--with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
--download-ctetgen  --download-triangle --with-debugging=yes
--download-netcdf  --download-zlib  --download-exodusii --download-hdf5
--download-pnetcdf

I'm trying to refine the DMPlexCreateBoxMesh  with the
DMPlexSetRefinementFunction.
It generally seems to be working, except that the refined element is
slightly offset from what I was expecting.
Based on my application specific criteria, element id number 7 is flagged
to be refined by the DMPlexSetRefinementFunction but when I open in
paraview, it looks like element id number 8 is being refined. See attached
pic.

[image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
[image: Screen Shot 2019-01-02 at 6.02.11 PM.png]

Is it possible that the maxVolumes array is 'off by one' when transfering
to tetgen data structures somehow ?

https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252



(gdb) bt
#0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
dmRefined=0x7fffffffb938) at
/opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
#1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
adaptLabel=0x0, dmRefined=0x7fffffffb938) at
/opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
#2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
dmRefined=0x7fffffffb938) at
/opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
#3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
dmf=0x7fffffffb938) at
/opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
#4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
<ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
/rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
#5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
/rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/fc293577/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-01-02 at 6.02.02 PM.png
Type: image/png
Size: 24717 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/fc293577/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-01-02 at 6.02.11 PM.png
Type: image/png
Size: 23658 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/fc293577/attachment-0003.png>

From jychang48 at gmail.com  Wed Jan  2 19:37:18 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 2 Jan 2019 18:37:18 -0700
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAMYG4GkDqg7LRG_pzVQ8dcofxw3pJRsT7urMjAtSsrjXenTPGg@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
	<CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
	<CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
	<CAMYG4GkDqg7LRG_pzVQ8dcofxw3pJRsT7urMjAtSsrjXenTPGg@mail.gmail.com>
Message-ID: <CAP2=TMjdQJLcL=ph_0sv_K1y9D6PXZtSnDK-=1ztZtbQo-5VhA@mail.gmail.com>

So I was slightly incorrect - these options I originally posted:

-fieldsplit_1_pc_fieldsplit_0_fields 3,4
-fieldsplit_1_pc_fieldsplit_1_fields 5
-fieldsplit_2_pc_fieldsplit_0_fields 6,7
-fieldsplit_2_pc_fieldsplit_1_fields 8

should instead be these:

-fieldsplit_1_pc_fieldsplit_0_fields 0,1
-fieldsplit_1_pc_fieldsplit_1_fields 2
-fieldsplit_2_pc_fieldsplit_0_fields 0,1
-fieldsplit_2_pc_fieldsplit_1_fields 2

So for a coupled two domain problem with three fields total: velocity,
pressure, and temperature where I'd want to schur complement the velocity
and pressure, I would use these options:

-ksp_monitor_true_residual
-ksp_view
-ksp_type gmres
-pc_type fieldsplit
-pc_fieldsplit_0_fields 0,1,2
-fieldsplit_0_ksp_type preonly
-fieldsplit_0_pc_type fieldsplit
-fieldsplit_0_pc_fieldsplit_type additive
-fieldsplit_0_pc_fieldsplit_0_fields 0,1
-fieldsplit_0_fieldsplit_0_pc_type fieldsplit
-fieldsplit_0_fieldsplit_0_pc_fieldsplit_type schur
-fieldsplit_0_fieldsplit_0_pc_fieldsplit_fact_type full
-fieldsplit_0_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
-fieldsplit_0_fieldsplit_0_fieldsplit_v1_ksp_type preonly
-fieldsplit_0_fieldsplit_0_fieldsplit_p1_ksp_type preonly
-fieldsplit_0_fieldsplit_0_fieldsplit_v1_pc_type bjacobi
-fieldsplit_0_fieldsplit_0_fieldsplit_p1_pc_type hypre
-fieldsplit_0_pc_fieldsplit_1_fields 2
-fieldsplit_0_fieldsplit_t1_ksp_type preonly
-fieldsplit_0_fieldsplit_t1_pc_type hypre
-pc_fieldsplit_1_fields 3,4,5
-fieldsplit_1_ksp_type preonly
-fieldsplit_1_pc_type fieldsplit
-fieldsplit_1_pc_fieldsplit_type additive
-fieldsplit_1_pc_fieldsplit_0_fields 0,1
-fieldsplit_1_fieldsplit_0_pc_type fieldsplit
-fieldsplit_1_fieldsplit_0_pc_fieldsplit_type schur
-fieldsplit_1_fieldsplit_0_pc_fieldsplit_fact_type full
-fieldsplit_1_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
-fieldsplit_1_fieldsplit_0_fieldsplit_v2_ksp_type preonly
-fieldsplit_1_fieldsplit_0_fieldsplit_p2_ksp_type preonly
-fieldsplit_1_fieldsplit_0_fieldsplit_v2_pc_type bjacobi
-fieldsplit_1_fieldsplit_0_fieldsplit_p2_pc_type hypre
-fieldsplit_1_pc_fieldsplit_1_fields 2
-fieldsplit_1_fieldsplit_t2_ksp_type preonly
-fieldsplit_1_fieldsplit_t2_pc_type hypre

Attached is the output of ksp_view

On Wed, Jan 2, 2019 at 3:39 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 2, 2019 at 4:55 PM Justin Chang <jychang48 at gmail.com> wrote:
>
>> Okay that's fine.
>>
>> Now consider one more case:
>>
>> Suppose I create a PetscSection with 9 fields:
>>
>> section = PETSc.Section().create()
>> section.setNumFields(9)
>> section.setFieldName(0,'u1')
>> section.setFieldName(1,'c1')
>> section.setFieldName(2,'t1')
>> section.setFieldName(3,'u2')
>> section.setFieldName(4,'c2')
>> section.setFieldName(5,'t2')
>> section.setFieldName(6,'u3')
>> section.setFieldName(7,'c3')
>> section.setFieldName(8,'t3')
>> section.setFieldComponents(0,1)
>> section.setFieldComponents(1,1)
>> section.setFieldComponents(2,1)
>> section.setFieldComponents(3,1)
>> section.setFieldComponents(4,1)
>> section.setFieldComponents(5,1)
>> section.setFieldComponents(6,1)
>> section.setFieldComponents(7,1)
>> section.setFieldComponents(8,1)
>>
>> ....
>>
>> where u is potential, c is concentration, and t is temperature. The
>> numbers refer to different domains of a battery (anode, electrolyte,
>> cathode).
>>
>> I want three levels of splits:
>> 1) Split by domain
>> 2) Split potential and concentration from temperature.
>> 3) Split potential and concentration for schur complementing purpose.
>>
>> Do the following PETSc command line options describe the split I
>> mentioned above:
>>
>> -pc_type fieldsplit
>> -pc_fieldsplit_0_fields 0,1,2
>> -pc_fieldsplit_1_fields 3,4,5
>> -pc_fieldsplit_2_fields 6,7,8
>> -fieldsplit_0_pc_type fieldsplit
>> -fieldsplit_1_pc_type fieldsplit
>> -fieldsplit_2_pc_type fieldsplit
>> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
>> -fieldsplit_0_pc_fieldsplit_1_fields 2
>> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
>> -fieldsplit_1_pc_fieldsplit_1_fields 5
>> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
>> -fieldsplit_2_pc_fieldsplit_1_fields 8
>>
>> I don't have an actual code to test these out with yet as I've never done
>> anything beyond two levels of split. So I am wondering if the above will
>> break anything as far as you can tell.
>>
>
> This should work as you expect. Send me the -ksp_view output when you run
> it. It would be very cool to see 3-level in action.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Justin
>>
>> On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com> wrote:
>>>
>>>> Okay so I managed to successfully do this by translating the IS's into
>>>> a PetscSection, assigning it to an empty DMShell, and then assigning that
>>>> to KSP.
>>>>
>>>> However, I seem to be unable to rename the outer fields that aggregate
>>>> two or more inner fields.
>>>>
>>>
>>> Right now you cannot do that because of the way that I check for the
>>> option:
>>>
>>>
>>> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>>>
>>> I guess we could replace this by some search code that looked for any
>>> option of that form and then read
>>> out the splitname using scanf. Right now, I do not see a pattern search
>>> function for the options database.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Consider this snippet of a four-field dual porosity/permeability
>>>> FEniCS/petsc4py code:
>>>>
>>>> ...
>>>>
>>>> ## Extract FEniCS dof layout, global indices ##
>>>> dof_total = np.array(W.dofmap().dofs())
>>>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>>>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>>>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>>>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>>>> offset = np.min(dof_total)
>>>>
>>>> ## Create PetscSection ##
>>>> section = PETSc.Section().create()
>>>> section.setNumFields(4)
>>>> section.setFieldName(0,'v1')
>>>> section.setFieldName(1,'p1')
>>>> section.setFieldName(2,'v2')
>>>> section.setFieldName(3,'p2')
>>>> section.setFieldComponents(0,1)
>>>> section.setFieldComponents(1,1)
>>>> section.setFieldComponents(2,1)
>>>> section.setFieldComponents(3,1)
>>>> section.setChart(0,len(dof_total))
>>>> for i in np.nditer(dof_v1):
>>>>   section.setDof(i-offset,1)
>>>>   section.setFieldDof(i-offset,0,1)
>>>> for i in np.nditer(dof_p1):
>>>>   section.setDof(i-offset,1)
>>>>   section.setFieldDof(i-offset,1,1)
>>>> for i in np.nditer(dof_v2):
>>>>   section.setDof(i-offset,1)
>>>>   section.setFieldDof(i-offset,2,1)
>>>> for i in np.nditer(dof_p2):
>>>>   section.setDof(i-offset,1)
>>>>   section.setFieldDof(i-offset,3,1)
>>>> section.setUp()
>>>>
>>>> ## Create DM and assign PetscSection ##
>>>> dm = PETSc.DMShell().create()
>>>> dm.setDefaultSection(section)
>>>> dm.setUp()
>>>>
>>>> ## Create KSP and assign DM ##
>>>> ksp = PETSc.KSP().create()
>>>> ksp.setDM(dm)
>>>> ksp.setDMActive(False)
>>>>
>>>> ### PETSc Command-line options ##
>>>> PETScOptions.set('ksp_monitor_true_residual')
>>>> PETScOptions.set('ksp_view')
>>>> PETScOptions.set('ksp_type','gmres')
>>>> PETScOptions.set('pc_type','fieldsplit')
>>>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>>>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>>>> PETScOptions.set('pc_fieldsplit_type','additive')
>>>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>>>>
>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>>>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>>>>
>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>>>
>>>> ...
>>>>
>>>> Is it possible to rename the outer splits (aka make
>>>> '-pc_fieldsplit_0_fields 0,1' be something like
>>>> '-pc_fieldsplit_micro_fields 0,1')?
>>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>>
>>>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am solving a six field battery problem (concentration and potential
>>>>>> for each of the two solid and one electrolyte domains) and I want to
>>>>>> experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>>>> this getting really messy from a programming standpoint, especially once I
>>>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>>>> rely on command line options to play around with various combinations of
>>>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>>>
>>>>>> However, it doesn't seem clear to me how I would create a DM when you
>>>>>> already know the IS's for each fields. If I understand functions like
>>>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>>>> the other way around. Perhaps the "reversal" of something like
>>>>>> DMCreateFieldIS()
>>>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>>>> way?
>>>>>>
>>>>>> Any thoughts/help appreciated!
>>>>>>
>>>>>
>>>>> Paul has recently done this for LibMesh. I believe that constructing a
>>>>> PetscSection is enough to get you minimally started. That allows
>>>>> DMCreateSubDM() to work by subsetting the Section, and that should
>>>>> allow the command line to work. CreateFieldDecomposition() should
>>>>> be removed I think.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> Justin
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/b45e48e3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output_6field_3nested_solves
Type: application/octet-stream
Size: 28028 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190102/b45e48e3/attachment-0001.obj>

From tempohoper at gmail.com  Thu Jan  3 04:56:52 2019
From: tempohoper at gmail.com (Sal Am)
Date: Thu, 3 Jan 2019 10:56:52 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
Message-ID: <CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=-22193

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=-24197

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=0

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=0

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[24]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=0

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[24]PETSC ERROR: #3 PCSetUp_LU() line 131 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 932 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 391 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=0

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 131 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 932 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 391 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=-35665

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[8]PETSC ERROR: #4 PCSetUp() line 932 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFOG(1)=-13, INFO(2)=-16661

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
3 04:32:43 2019
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
--with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=no --download-scalapack --download-superlu
--download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #6 main() line 68 in
/lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -ksp_error_if_not_converged
[24]PETSC ERROR: -ksp_max_it 1
[24]PETSC ERROR: -ksp_monitor_true_residual
[24]PETSC ERROR: -ksp_type richardson
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_mumps_icntl_14 100
[24]PETSC ERROR: -pc_factor_mat_solver_type mumps
[24]PETSC ERROR: -pc_type lu
[24]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------

Sorry to bother again, but increasing ICNTL(14) does not seem to have
helped... I am not sure what I am doing wrong exactly. As you can see I
have increased it by 100%, prior, I tried to increase it with 40-70% but
none worked.

On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

>
>
> On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi I am getting an error using MUMPS.
>> How I run it:
>> bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu
>> -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual
>> -log_view -ksp_error_if_not_converged
>>
>> The error output:
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Error in external library
>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [1]PETSC ERROR: Error in external library
>> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [6]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [6]PETSC ERROR: Error in external library
>> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [8]PETSC ERROR: Error in external library
>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: ---------------------
>> Error Message --------------------------------------------------------------
>> [15]PETSC ERROR: Error in external library
>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=-36536
>>
>> [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=-81813
>>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: Error in external library
>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [3]PETSC ERROR: Error in external library
>> [3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=-28194
>>
>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [4]PETSC ERROR: Error in external library
>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [5]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [5]PETSC ERROR: Error in external library
>> [5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=-27685
>>
>> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [7]PETSC ERROR: Error in external library
>> [7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFOG(1)=-13, INFO(2)=0
>>
>> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec
>> 12 10:44:13 2018
>> [7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>> ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>> [10]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [10]PETSC ERROR: Error in external library
>> [10]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=0
>>
>> [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [11]PETSC ERROR: Error in external library
>> [11]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=-30519
>>
>> [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [12]PETSC ERROR: Error in external library
>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=-28435
>>
>> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [13]PETSC ERROR: Error in external library
>> [13]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=-30110
>>
>> [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [14]PETSC ERROR: Error in external library
>> [14]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFOG(1)=-13, INFO(2)=0
>>
>> [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>> [14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>> Dec 12 10:44:13 2018
>> [14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>> --with-debugging=no --download-scalapack --download-superlu
>> --download-mpich --download-fblaslapack=1 --download-cmake
>> [14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>> #1 MatFactorNumeric_MUMPS() line 1230 in
>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>> [15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>> [15]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>> [15]PETSC ERROR: #4 PCSetUp() line 932 in
>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>> [15]PETSC ERROR: #5 KSPSetUp() line 391 in
>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>> [15]PETSC ERROR: #6 main() line 68 in
>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>> [15]PETSC ERROR: PETSc Option Table entries:
>> [15]PETSC ERROR: -ksp_error_if_not_converged
>> [15]PETSC ERROR: -ksp_max_it 1
>> [15]PETSC ERROR: -ksp_monitor_true_residual
>> [15]PETSC ERROR: -ksp_type richardson
>> [15]PETSC ERROR: -log_view
>> [15]PETSC ERROR: -pc_factor_mat_solver_type mumps
>> [15]PETSC ERROR: -pc_type lu
>> [15]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>>
>> From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
>> Problem of workspace allocation of size INFO(2) during the factorization
>> or solve steps. The size that the package tried to allocate with a Fortran
>> ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then
>> the size that the package requested is obtained by multiplying the absolute
>> value of INFO(2) by 1 million. In general, the unit for INFO(2) is the
>> number of scalar entries of  the type of the input matrix (real, complex,
>> single or double precision).
>>
>> Not quite sure what to make of it as the first error indicates size
>> INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
>>
> But you can see there are processes with negative INFO(2). According to
> MUMPS manual, you can try to increase ICNTL(14).
>
>
>>
>> Thanks
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/8ba678d0/attachment-0001.html>

From yjwu16 at gmail.com  Thu Jan  3 06:34:50 2019
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Thu, 3 Jan 2019 07:34:50 -0500
Subject: [petsc-users] Problems about Picard and NolinearGS
In-Reply-To: <87ftuj2ef3.fsf@jedbrown.org>
References: <CAAMEhBjJKFAthSu11GKna7id6EHXTPHGuOhT-somy_=ST8_TAA@mail.gmail.com>
	<FFF11830-A669-496F-A750-EC009A463B84@anl.gov>
	<CAAMEhBhszto1xLSy7gqbM+ujjhWnKfdK-7wAQMRgsKhn_A9BJw@mail.gmail.com>
	<7BDB5EF2-6C9C-4AFD-A0AB-4C127596A174@mcs.anl.gov>
	<CAAMEhBjff8nrRJzo6n8W=8KkvL4THcRqWNqUPC3GBi7+vp3iBg@mail.gmail.com>
	<87ftuj2ef3.fsf@jedbrown.org>
Message-ID: <CAAMEhBg9AViodmp3Oot2BJ7BXCxKmxTwYTfXTWDgM2zs0c+mdA@mail.gmail.com>

Thanks for your reply.
I read the article you provided. This is my first contact with the
quasi-Newton method.
I have some problem:
1. From the point of view of algorithm, the quasi-Newton method does not
need to be solved by linear equations, so why would KSP be used?
2. Where and how to  use preconditioner in quasi-Newton method?

Thanks,
Yingjie

Jed Brown <jed at jedbrown.org> ?2018?12?27??? ??10:11???

> Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > I my opinion, the difficulty in constructing my Jacobian matrix is
> complex
> > coefficient.(eg, thermal conductivity* ?* , density )
> > For example, in temperature equation(T):
> >                 ?(*?*?T) - ?(?* Cp* u ) + Q =  0
> > *?* is thermal conductivity , ?* is density  Cp* is specific heat , u is
> > velocity, Q is source.
> > *? = *1.9*(1.0e-3)*pow(T+273.0-150.0,1.29)            function of T
> > ?=
> >
> (0.4814*P/1.0e3/(T+273.15))/(1.0+0.446*(1.0e-2)*P/1.0e3/(pow(T+273.15,1.2)))
> > function of T and P
> >
> > In theory, the coefficient contain variable. So it is complicated to
> > calculate the element of Jacobian.
> > In my snes_mf_operator method, I treat ?,? as constant. In every
> nonlinear
> > step, I use the solution update the ?,?  and thus update the
> > preconditioning matrix. At each residual function call(in
> > SNESFormFunction), I also update the coefficient to ensure the correction
> > of the residual function.
>
> If those Jacobian entries are really that hard to get right, you can try
> using quasi-Newton as an alternative to -snes_mf_operator; similar to
>
>
> https://jedbrown.org/files/BrownBrune-LowRankQuasiNewtonRobustJacobianLagging-2013.pdf
>
>
> In general, I would start with an exact Jacobian (computed by coloring,
> AD, or just in one specific material/configuration).  Test Newton using
> direct solvers to see your "best case" convergence (globalization may be
> needed).  Test fieldsplit using direct solvers on the blocks so you know
> how much error is attributable to that approximation.  Only then look at
> backing off on the approximation of the Jacobian and the preconditioner.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/6ef8b850/attachment.html>

From knepley at gmail.com  Thu Jan  3 07:35:51 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 08:35:51 -0500
Subject: [petsc-users] Problems about Picard and NolinearGS
In-Reply-To: <CAAMEhBg9AViodmp3Oot2BJ7BXCxKmxTwYTfXTWDgM2zs0c+mdA@mail.gmail.com>
References: <CAAMEhBjJKFAthSu11GKna7id6EHXTPHGuOhT-somy_=ST8_TAA@mail.gmail.com>
	<FFF11830-A669-496F-A750-EC009A463B84@anl.gov>
	<CAAMEhBhszto1xLSy7gqbM+ujjhWnKfdK-7wAQMRgsKhn_A9BJw@mail.gmail.com>
	<7BDB5EF2-6C9C-4AFD-A0AB-4C127596A174@mcs.anl.gov>
	<CAAMEhBjff8nrRJzo6n8W=8KkvL4THcRqWNqUPC3GBi7+vp3iBg@mail.gmail.com>
	<87ftuj2ef3.fsf@jedbrown.org>
	<CAAMEhBg9AViodmp3Oot2BJ7BXCxKmxTwYTfXTWDgM2zs0c+mdA@mail.gmail.com>
Message-ID: <CAMYG4GkWUwgdX8PYAP-f4dqBgeKchq6b=bHRqpRyvF_uyqEWBg@mail.gmail.com>

On Thu, Jan 3, 2019 at 7:36 AM Yingjie Wu via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thanks for your reply.
> I read the article you provided. This is my first contact with the
> quasi-Newton method.
> I have some problem:
> 1. From the point of view of algorithm, the quasi-Newton method does not
> need to be solved by linear equations, so why would KSP be used?
>

It is solving the equations, but we know the analytical answer. KSP is our
abstraction for linear equation solver,
so we also use it in this case.


> 2. Where and how to  use preconditioner in quasi-Newton method?
>

You do not need a PC here. Note that this is only going to work well if you
have a good initial
guess for the inverse of your Jacobian. The optimization people who
invented have that (it is
the identity). In Jed's paper, they use a V-cycle as the good initial guess.

  Thanks,

    Matt


> Thanks,
> Yingjie
>
> Jed Brown <jed at jedbrown.org> ?2018?12?27??? ??10:11???
>
>> Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov> writes:
>>
>> > I my opinion, the difficulty in constructing my Jacobian matrix is
>> complex
>> > coefficient.(eg, thermal conductivity* ?* , density )
>> > For example, in temperature equation(T):
>> >                 ?(*?*?T) - ?(?* Cp* u ) + Q =  0
>> > *?* is thermal conductivity , ?* is density  Cp* is specific heat , u is
>> > velocity, Q is source.
>> > *? = *1.9*(1.0e-3)*pow(T+273.0-150.0,1.29)            function of T
>> > ?=
>> >
>> (0.4814*P/1.0e3/(T+273.15))/(1.0+0.446*(1.0e-2)*P/1.0e3/(pow(T+273.15,1.2)))
>> > function of T and P
>> >
>> > In theory, the coefficient contain variable. So it is complicated to
>> > calculate the element of Jacobian.
>> > In my snes_mf_operator method, I treat ?,? as constant. In every
>> nonlinear
>> > step, I use the solution update the ?,?  and thus update the
>> > preconditioning matrix. At each residual function call(in
>> > SNESFormFunction), I also update the coefficient to ensure the
>> correction
>> > of the residual function.
>>
>> If those Jacobian entries are really that hard to get right, you can try
>> using quasi-Newton as an alternative to -snes_mf_operator; similar to
>>
>>
>> https://jedbrown.org/files/BrownBrune-LowRankQuasiNewtonRobustJacobianLagging-2013.pdf
>>
>>
>> In general, I would start with an exact Jacobian (computed by coloring,
>> AD, or just in one specific material/configuration).  Test Newton using
>> direct solvers to see your "best case" convergence (globalization may be
>> needed).  Test fieldsplit using direct solvers on the blocks so you know
>> how much error is attributable to that approximation.  Only then look at
>> backing off on the approximation of the Jacobian and the preconditioner.
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/4418a8ef/attachment.html>

From knepley at gmail.com  Thu Jan  3 07:37:52 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 08:37:52 -0500
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
Message-ID: <CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>

On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=-22193
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
> 3 04:32:43 2019
> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [4]PETSC ERROR: Error in external library
> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=-24197
>
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
> 3 04:32:43 2019
> [4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [16]PETSC ERROR: Error in external library
> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=0
>
> [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
> Jan  3 04:32:43 2019
> [16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [20]PETSC ERROR: Error in external library
> [20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=0
>
> [20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
> Jan  3 04:32:43 2019
> [20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [24]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [24]PETSC ERROR: Error in external library
> [24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=0
>
> [24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
> Jan  3 04:32:43 2019
> [24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [24]PETSC ERROR: #3 PCSetUp_LU() line 131 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [24]PETSC ERROR: #4 PCSetUp() line 932 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [24]PETSC ERROR: #5 KSPSetUp() line 391 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [32]PETSC ERROR: Error in external library
> [32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=0
>
> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
> Jan  3 04:32:43 2019
> [32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [32]PETSC ERROR: #3 PCSetUp_LU() line 131 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [32]PETSC ERROR: #4 PCSetUp() line 932 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [32]PETSC ERROR: #5 KSPSetUp() line 391 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [8]PETSC ERROR: Error in external library
> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=-35665
>
> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan
> 3 04:32:43 2019
> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [8]PETSC ERROR: #4 PCSetUp() line 932 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [12]PETSC ERROR: Error in external library
> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFOG(1)=-13, INFO(2)=-16661
>
> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
> Jan  3 04:32:43 2019
> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-mpich --download-fblaslapack=1 --download-cmake
> [12]PETSC ERROR: #6 main() line 68 in
> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
> [24]PETSC ERROR: PETSc Option Table entries:
> [24]PETSC ERROR: -ksp_error_if_not_converged
> [24]PETSC ERROR: -ksp_max_it 1
> [24]PETSC ERROR: -ksp_monitor_true_residual
> [24]PETSC ERROR: -ksp_type richardson
> [24]PETSC ERROR: -log_view
> [24]PETSC ERROR: -mat_mumps_icntl_14 100
> [24]PETSC ERROR: -pc_factor_mat_solver_type mumps
> [24]PETSC ERROR: -pc_type lu
> [24]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
>
> Sorry to bother again, but increasing ICNTL(14) does not seem to have
> helped... I am not sure what I am doing wrong exactly. As you can see I
> have increased it by 100%, prior, I tried to increase it with 40-70% but
> none worked.
>

I think this is for the MUMPS list, but it appears you have a lot of
fill-in for this matrix and it might be too much for a direct solver.

  Thanks,

    Matt


> On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov>
> wrote:
>
>>
>>
>> On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi I am getting an error using MUMPS.
>>> How I run it:
>>> bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu
>>> -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual
>>> -log_view -ksp_error_if_not_converged
>>>
>>> The error output:
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Error in external library
>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [1]PETSC ERROR: Error in external library
>>> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [6]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [6]PETSC ERROR: Error in external library
>>> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [8]PETSC ERROR: Error in external library
>>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: ---------------------
>>> Error Message --------------------------------------------------------------
>>> [15]PETSC ERROR: Error in external library
>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-36536
>>>
>>> [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-81813
>>>
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [2]PETSC ERROR: Error in external library
>>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [3]PETSC ERROR: Error in external library
>>> [3]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-28194
>>>
>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [4]PETSC ERROR: Error in external library
>>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [5]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [5]PETSC ERROR: Error in external library
>>> [5]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-27685
>>>
>>> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [7]PETSC ERROR: Error in external library
>>> [7]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13
>>> 2018
>>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>> [10]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [10]PETSC ERROR: Error in external library
>>> [10]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [11]PETSC ERROR: Error in external library
>>> [11]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-30519
>>>
>>> [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [12]PETSC ERROR: Error in external library
>>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-28435
>>>
>>> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [13]PETSC ERROR: Error in external library
>>> [13]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-30110
>>>
>>> [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [14]PETSC ERROR: Error in external library
>>> [14]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>> Dec 12 10:44:13 2018
>>> [14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [15]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>> [15]PETSC ERROR: #4 PCSetUp() line 932 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [15]PETSC ERROR: #5 KSPSetUp() line 391 in
>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>> [15]PETSC ERROR: #6 main() line 68 in
>>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>>> [15]PETSC ERROR: PETSc Option Table entries:
>>> [15]PETSC ERROR: -ksp_error_if_not_converged
>>> [15]PETSC ERROR: -ksp_max_it 1
>>> [15]PETSC ERROR: -ksp_monitor_true_residual
>>> [15]PETSC ERROR: -ksp_type richardson
>>> [15]PETSC ERROR: -log_view
>>> [15]PETSC ERROR: -pc_factor_mat_solver_type mumps
>>> [15]PETSC ERROR: -pc_type lu
>>> [15]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>>
>>> From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
>>> Problem of workspace allocation of size INFO(2) during the factorization
>>> or solve steps. The size that the package tried to allocate with a Fortran
>>> ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then
>>> the size that the package requested is obtained by multiplying the absolute
>>> value of INFO(2) by 1 million. In general, the unit for INFO(2) is the
>>> number of scalar entries of  the type of the input matrix (real, complex,
>>> single or double precision).
>>>
>>> Not quite sure what to make of it as the first error indicates size
>>> INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
>>>
>> But you can see there are processes with negative INFO(2). According to
>> MUMPS manual, you can try to increase ICNTL(14).
>>
>>
>>>
>>> Thanks
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/cf66a41f/attachment-0001.html>

From knepley at gmail.com  Thu Jan  3 07:49:27 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 08:49:27 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
Message-ID: <CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>

On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> Starting with TS example 45
>
> $ pwd
> /opt/apps/PETSc/petsc-3.10.2
> $ ls src/ts/examples/tutorials/ex45.c
> src/ts/examples/tutorials/ex45.c
>
> petsc configured with: ./config/configure.py --with-shared-libraries
> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
> --download-ctetgen  --download-triangle --with-debugging=yes
> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
> --download-pnetcdf
>
> I'm trying to refine the DMPlexCreateBoxMesh  with the
> DMPlexSetRefinementFunction.
> It generally seems to be working, except that the refined element is
> slightly offset from what I was expecting.
> Based on my application specific criteria, element id number 7 is flagged
> to be refined by the DMPlexSetRefinementFunction but when I open in
> paraview, it looks like element id number 8 is being refined. See attached
> pic.
>
> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>
> Is it possible that the maxVolumes array is 'off by one' when transfering
> to tetgen data structures somehow ?
>

I looked through and cannot see it by eye. Could you send me your modified
example and I will walk through it with the
debugger?

  Thanks,

     Matt


>
> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>
>
>
> (gdb) bt
> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
> dmRefined=0x7fffffffb938) at
> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
> dmRefined=0x7fffffffb938) at
> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
> dmf=0x7fffffffb938) at
> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/11ddcf63/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-01-02 at 6.02.02 PM.png
Type: image/png
Size: 24717 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/11ddcf63/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-01-02 at 6.02.11 PM.png
Type: image/png
Size: 23658 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/11ddcf63/attachment-0003.png>

From knepley at gmail.com  Thu Jan  3 08:04:23 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 09:04:23 -0500
Subject: [petsc-users] Create a DM given sets of IS's
In-Reply-To: <CAP2=TMjdQJLcL=ph_0sv_K1y9D6PXZtSnDK-=1ztZtbQo-5VhA@mail.gmail.com>
References: <CAP2=TMgjwmPvu4j=XjJdizy924oTi_uibN=KmGANB=vs2p8k0A@mail.gmail.com>
	<CAMYG4GmV=HJ3nUA1eJH6a4sDnyWz-+OgyW9pf_whfxHtb-9LDg@mail.gmail.com>
	<CAP2=TMissKUQPK8hfgiOW9TFd7MLYxXD8sh8TxKOFumRD-5AoA@mail.gmail.com>
	<CAMYG4Gnf1Yi=aGc4O4ZV8+STQo6erX2o84ueXgnH3qHOf_dr+A@mail.gmail.com>
	<CAP2=TMhpzV-pXxNiUEuNSMEJEF0PfRL4grRGF=SE2WzFCXpMZA@mail.gmail.com>
	<CAMYG4GkDqg7LRG_pzVQ8dcofxw3pJRsT7urMjAtSsrjXenTPGg@mail.gmail.com>
	<CAP2=TMjdQJLcL=ph_0sv_K1y9D6PXZtSnDK-=1ztZtbQo-5VhA@mail.gmail.com>
Message-ID: <CAMYG4GkfGZSOjpqX19pFoBGotr2cuAqm5aGBKBKDG3vZ30VW2w@mail.gmail.com>

Looks like it is converging great. There are two things:

- The inner FS is additive, which seems to work fine here, but maybe not
when there is more coupling with temperature
- The BJacobi/ILU might not scale well.

  Thanks,

     Matt

On Wed, Jan 2, 2019 at 8:38 PM Justin Chang <jychang48 at gmail.com> wrote:

> So I was slightly incorrect - these options I originally posted:
>
> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
> -fieldsplit_1_pc_fieldsplit_1_fields 5
> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
> -fieldsplit_2_pc_fieldsplit_1_fields 8
>
> should instead be these:
>
> -fieldsplit_1_pc_fieldsplit_0_fields 0,1
> -fieldsplit_1_pc_fieldsplit_1_fields 2
> -fieldsplit_2_pc_fieldsplit_0_fields 0,1
> -fieldsplit_2_pc_fieldsplit_1_fields 2
>
> So for a coupled two domain problem with three fields total: velocity,
> pressure, and temperature where I'd want to schur complement the velocity
> and pressure, I would use these options:
>
> -ksp_monitor_true_residual
> -ksp_view
> -ksp_type gmres
> -pc_type fieldsplit
> -pc_fieldsplit_0_fields 0,1,2
> -fieldsplit_0_ksp_type preonly
> -fieldsplit_0_pc_type fieldsplit
> -fieldsplit_0_pc_fieldsplit_type additive
> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
> -fieldsplit_0_fieldsplit_0_pc_type fieldsplit
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_type schur
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_fact_type full
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
> -fieldsplit_0_fieldsplit_0_fieldsplit_v1_ksp_type preonly
> -fieldsplit_0_fieldsplit_0_fieldsplit_p1_ksp_type preonly
> -fieldsplit_0_fieldsplit_0_fieldsplit_v1_pc_type bjacobi
> -fieldsplit_0_fieldsplit_0_fieldsplit_p1_pc_type hypre
> -fieldsplit_0_pc_fieldsplit_1_fields 2
> -fieldsplit_0_fieldsplit_t1_ksp_type preonly
> -fieldsplit_0_fieldsplit_t1_pc_type hypre
> -pc_fieldsplit_1_fields 3,4,5
> -fieldsplit_1_ksp_type preonly
> -fieldsplit_1_pc_type fieldsplit
> -fieldsplit_1_pc_fieldsplit_type additive
> -fieldsplit_1_pc_fieldsplit_0_fields 0,1
> -fieldsplit_1_fieldsplit_0_pc_type fieldsplit
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_type schur
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_fact_type full
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
> -fieldsplit_1_fieldsplit_0_fieldsplit_v2_ksp_type preonly
> -fieldsplit_1_fieldsplit_0_fieldsplit_p2_ksp_type preonly
> -fieldsplit_1_fieldsplit_0_fieldsplit_v2_pc_type bjacobi
> -fieldsplit_1_fieldsplit_0_fieldsplit_p2_pc_type hypre
> -fieldsplit_1_pc_fieldsplit_1_fields 2
> -fieldsplit_1_fieldsplit_t2_ksp_type preonly
> -fieldsplit_1_fieldsplit_t2_pc_type hypre
>
> Attached is the output of ksp_view
>
> On Wed, Jan 2, 2019 at 3:39 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Jan 2, 2019 at 4:55 PM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Okay that's fine.
>>>
>>> Now consider one more case:
>>>
>>> Suppose I create a PetscSection with 9 fields:
>>>
>>> section = PETSc.Section().create()
>>> section.setNumFields(9)
>>> section.setFieldName(0,'u1')
>>> section.setFieldName(1,'c1')
>>> section.setFieldName(2,'t1')
>>> section.setFieldName(3,'u2')
>>> section.setFieldName(4,'c2')
>>> section.setFieldName(5,'t2')
>>> section.setFieldName(6,'u3')
>>> section.setFieldName(7,'c3')
>>> section.setFieldName(8,'t3')
>>> section.setFieldComponents(0,1)
>>> section.setFieldComponents(1,1)
>>> section.setFieldComponents(2,1)
>>> section.setFieldComponents(3,1)
>>> section.setFieldComponents(4,1)
>>> section.setFieldComponents(5,1)
>>> section.setFieldComponents(6,1)
>>> section.setFieldComponents(7,1)
>>> section.setFieldComponents(8,1)
>>>
>>> ....
>>>
>>> where u is potential, c is concentration, and t is temperature. The
>>> numbers refer to different domains of a battery (anode, electrolyte,
>>> cathode).
>>>
>>> I want three levels of splits:
>>> 1) Split by domain
>>> 2) Split potential and concentration from temperature.
>>> 3) Split potential and concentration for schur complementing purpose.
>>>
>>> Do the following PETSc command line options describe the split I
>>> mentioned above:
>>>
>>> -pc_type fieldsplit
>>> -pc_fieldsplit_0_fields 0,1,2
>>> -pc_fieldsplit_1_fields 3,4,5
>>> -pc_fieldsplit_2_fields 6,7,8
>>> -fieldsplit_0_pc_type fieldsplit
>>> -fieldsplit_1_pc_type fieldsplit
>>> -fieldsplit_2_pc_type fieldsplit
>>> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
>>> -fieldsplit_0_pc_fieldsplit_1_fields 2
>>> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
>>> -fieldsplit_1_pc_fieldsplit_1_fields 5
>>> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
>>> -fieldsplit_2_pc_fieldsplit_1_fields 8
>>>
>>> I don't have an actual code to test these out with yet as I've never
>>> done anything beyond two levels of split. So I am wondering if the above
>>> will break anything as far as you can tell.
>>>
>>
>> This should work as you expect. Send me the -ksp_view output when you run
>> it. It would be very cool to see 3-level in action.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> Okay so I managed to successfully do this by translating the IS's into
>>>>> a PetscSection, assigning it to an empty DMShell, and then assigning that
>>>>> to KSP.
>>>>>
>>>>> However, I seem to be unable to rename the outer fields that aggregate
>>>>> two or more inner fields.
>>>>>
>>>>
>>>> Right now you cannot do that because of the way that I check for the
>>>> option:
>>>>
>>>>
>>>> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>>>>
>>>> I guess we could replace this by some search code that looked for any
>>>> option of that form and then read
>>>> out the splitname using scanf. Right now, I do not see a pattern search
>>>> function for the options database.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Consider this snippet of a four-field dual porosity/permeability
>>>>> FEniCS/petsc4py code:
>>>>>
>>>>> ...
>>>>>
>>>>> ## Extract FEniCS dof layout, global indices ##
>>>>> dof_total = np.array(W.dofmap().dofs())
>>>>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>>>>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>>>>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>>>>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>>>>> offset = np.min(dof_total)
>>>>>
>>>>> ## Create PetscSection ##
>>>>> section = PETSc.Section().create()
>>>>> section.setNumFields(4)
>>>>> section.setFieldName(0,'v1')
>>>>> section.setFieldName(1,'p1')
>>>>> section.setFieldName(2,'v2')
>>>>> section.setFieldName(3,'p2')
>>>>> section.setFieldComponents(0,1)
>>>>> section.setFieldComponents(1,1)
>>>>> section.setFieldComponents(2,1)
>>>>> section.setFieldComponents(3,1)
>>>>> section.setChart(0,len(dof_total))
>>>>> for i in np.nditer(dof_v1):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,0,1)
>>>>> for i in np.nditer(dof_p1):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,1,1)
>>>>> for i in np.nditer(dof_v2):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,2,1)
>>>>> for i in np.nditer(dof_p2):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,3,1)
>>>>> section.setUp()
>>>>>
>>>>> ## Create DM and assign PetscSection ##
>>>>> dm = PETSc.DMShell().create()
>>>>> dm.setDefaultSection(section)
>>>>> dm.setUp()
>>>>>
>>>>> ## Create KSP and assign DM ##
>>>>> ksp = PETSc.KSP().create()
>>>>> ksp.setDM(dm)
>>>>> ksp.setDMActive(False)
>>>>>
>>>>> ### PETSc Command-line options ##
>>>>> PETScOptions.set('ksp_monitor_true_residual')
>>>>> PETScOptions.set('ksp_view')
>>>>> PETScOptions.set('ksp_type','gmres')
>>>>> PETScOptions.set('pc_type','fieldsplit')
>>>>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>>>>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>>>>> PETScOptions.set('pc_fieldsplit_type','additive')
>>>>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>>>>>
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>>>>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>>>>>
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>>>>
>>>>> ...
>>>>>
>>>>> Is it possible to rename the outer splits (aka make
>>>>> '-pc_fieldsplit_0_fields 0,1' be something like
>>>>> '-pc_fieldsplit_micro_fields 0,1')?
>>>>>
>>>>> Thanks,
>>>>> Justin
>>>>>
>>>>>
>>>>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am solving a six field battery problem (concentration and
>>>>>>> potential for each of the two solid and one electrolyte domains) and I want
>>>>>>> to experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>>>>> this getting really messy from a programming standpoint, especially once I
>>>>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>>>>> rely on command line options to play around with various combinations of
>>>>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>>>>
>>>>>>> However, it doesn't seem clear to me how I would create a DM when
>>>>>>> you already know the IS's for each fields. If I understand functions like
>>>>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>>>>> the other way around. Perhaps the "reversal" of something like
>>>>>>> DMCreateFieldIS()
>>>>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>>>>> way?
>>>>>>>
>>>>>>> Any thoughts/help appreciated!
>>>>>>>
>>>>>>
>>>>>> Paul has recently done this for LibMesh. I believe that constructing
>>>>>> a PetscSection is enough to get you minimally started. That allows
>>>>>> DMCreateSubDM() to work by subsetting the Section, and that should
>>>>>> allow the command line to work. CreateFieldDecomposition() should
>>>>>> be removed I think.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> Justin
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/f8b2ac02/attachment.html>

From hzhang at mcs.anl.gov  Thu Jan  3 09:33:27 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 3 Jan 2019 15:33:27 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
Message-ID: <CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>

You may try different matrix orderings, or try superlu_dist.
Hong


On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-22193

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-24197

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[24]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-35665

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[8]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-16661

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -ksp_error_if_not_converged
[24]PETSC ERROR: -ksp_max_it 1
[24]PETSC ERROR: -ksp_monitor_true_residual
[24]PETSC ERROR: -ksp_type richardson
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_mumps_icntl_14 100
[24]PETSC ERROR: -pc_factor_mat_solver_type mumps
[24]PETSC ERROR: -pc_type lu
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------

Sorry to bother again, but increasing ICNTL(14) does not seem to have helped... I am not sure what I am doing wrong exactly. As you can see I have increased it by 100%, prior, I tried to increase it with 40-70% but none worked.

I think this is for the MUMPS list, but it appears you have a lot of fill-in for this matrix and it might be too much for a direct solver.

  Thanks,

    Matt

On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:


On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi I am getting an error using MUMPS.
How I run it:
bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

The error output:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-36536

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-81813

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28194

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-27685

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30519

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28435

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30110

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
#1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[15]PETSC ERROR: PETSc Option Table entries:
[15]PETSC ERROR: -ksp_error_if_not_converged
[15]PETSC ERROR: -ksp_max_it 1
[15]PETSC ERROR: -ksp_monitor_true_residual
[15]PETSC ERROR: -ksp_type richardson
[15]PETSC ERROR: -log_view
[15]PETSC ERROR: -pc_factor_mat_solver_type mumps
[15]PETSC ERROR: -pc_type lu
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------

From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
Problem of workspace allocation of size INFO(2) during the factorization or solve steps. The size that the package tried to allocate with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then the size that the package requested is obtained by multiplying the absolute value of INFO(2) by 1 million. In general, the unit for INFO(2) is the number of scalar entries of  the type of the input matrix (real, complex, single or double precision).

Not quite sure what to make of it as the first error indicates size INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
But you can see there are processes with negative INFO(2). According to MUMPS manual, you can try to increase ICNTL(14).


Thanks


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/ad630cb0/attachment-0001.html>

From kandanovian at gmail.com  Thu Jan  3 09:38:37 2019
From: kandanovian at gmail.com (Tim Steinhoff)
Date: Thu, 3 Jan 2019 16:38:37 +0100
Subject: [petsc-users] definition of level in PCILU
Message-ID: <CAE=bjn5OCENjEya-cUPgeBE5sjbpEqsOs7fg-kPH9=2k0YJGpQ@mail.gmail.com>

Dear PETSc Team

I checked the references that are stated on

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html

to get an idea of the definition of <level> that is used in PETSc's
ILU(k) implementation. I haven't found anything of substance in the
first reference (T. Dupont, R. Kendall, and H. Rachford) and only a
rather vague description in the third one (Sec. 3.1 in the book). The
second one isn't yet available to me.

Additionally, I checked

https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf

which gives a thorough explanation in Subsec. 10.3.3. Also, I took a look at

https://www.cs.odu.edu/~pothen/Papers/ilu-levels.pdf

which gives _two_ different definitions of level (p. 3 'sum' and
'max'). The sum-version equals the one in Saad. The statement on
Wikipedia defines level according to the nonzero pattern of powers of
A.

So the question is: Which definition of <level> was utilized for the
implementation in PETSc? Any of the above? Or something else?

Thanks and kind regards
tim

P.S.: Happy New Year!

From stefano.zampini at gmail.com  Thu Jan  3 09:55:46 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 3 Jan 2019 16:55:46 +0100
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
	<CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
Message-ID: <DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>

Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using

 -mat_mumps_icntl_13 1

> On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> You may try different matrix orderings, or try superlu_dist.
> Hong
> 
> 
> On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-22193
> 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [4]PETSC ERROR: Error in external library
> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-24197
> 
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [16]PETSC ERROR: Error in external library
> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [16]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [20]PETSC ERROR: Error in external library
> [20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [20]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [24]PETSC ERROR: Error in external library
> [24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [24]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [24]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [24]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [24]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [32]PETSC ERROR: Error in external library
> [32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [32]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [32]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [32]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [32]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [8]PETSC ERROR: Error in external library
> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-35665
> 
> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [8]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [12]PETSC ERROR: Error in external library
> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-16661
> 
> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [12]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
> [24]PETSC ERROR: PETSc Option Table entries:
> [24]PETSC ERROR: -ksp_error_if_not_converged
> [24]PETSC ERROR: -ksp_max_it 1
> [24]PETSC ERROR: -ksp_monitor_true_residual
> [24]PETSC ERROR: -ksp_type richardson
> [24]PETSC ERROR: -log_view
> [24]PETSC ERROR: -mat_mumps_icntl_14 100
> [24]PETSC ERROR: -pc_factor_mat_solver_type mumps
> [24]PETSC ERROR: -pc_type lu
> [24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 
> Sorry to bother again, but increasing ICNTL(14) does not seem to have helped... I am not sure what I am doing wrong exactly. As you can see I have increased it by 100%, prior, I tried to increase it with 40-70% but none worked. 
> 
> I think this is for the MUMPS list, but it appears you have a lot of fill-in for this matrix and it might be too much for a direct solver.
> 
>   Thanks,
> 
>     Matt
>  
> On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov <mailto:jczhang at mcs.anl.gov>> wrote:
> 
> 
> On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> Hi I am getting an error using MUMPS. 
> How I run it:
> bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
> 
> The error output: 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Error in external library
> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [6]PETSC ERROR: Error in external library
> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [8]PETSC ERROR: Error in external library
> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [15]PETSC ERROR: Error in external library
> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-36536
> 
> [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [15]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-81813
> 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [1]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: Error in external library
> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [3]PETSC ERROR: Error in external library
> [3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28194
> 
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [3]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [4]PETSC ERROR: Error in external library
> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [5]PETSC ERROR: Error in external library
> [5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-27685
> 
> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [5]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [7]PETSC ERROR: Error in external library
> [7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [7]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [10]PETSC ERROR: Error in external library
> [10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [11]PETSC ERROR: Error in external library
> [11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30519
> 
> [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [11]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [12]PETSC ERROR: Error in external library
> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28435
> 
> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [13]PETSC ERROR: Error in external library
> [13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30110
> 
> [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [13]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [14]PETSC ERROR: Error in external library
> [14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0
> 
> [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
> [14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
> [14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
> [15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [15]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
> [15]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [15]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [15]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
> [15]PETSC ERROR: PETSc Option Table entries:
> [15]PETSC ERROR: -ksp_error_if_not_converged
> [15]PETSC ERROR: -ksp_max_it 1
> [15]PETSC ERROR: -ksp_monitor_true_residual
> [15]PETSC ERROR: -ksp_type richardson
> [15]PETSC ERROR: -log_view
> [15]PETSC ERROR: -pc_factor_mat_solver_type mumps
> [15]PETSC ERROR: -pc_type lu
> [15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 
> From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
> Problem of workspace allocation of size INFO(2) during the factorization or solve steps. The size that the package tried to allocate with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then the size that the package requested is obtained by multiplying the absolute value of INFO(2) by 1 million. In general, the unit for INFO(2) is the number of scalar entries of  the type of the input matrix (real, complex, single or double precision).
> 
> Not quite sure what to make of it as the first error indicates size INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
> But you can see there are processes with negative INFO(2). According to MUMPS manual, you can try to increase ICNTL(14).
>  
> 
> Thanks
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/b22f4f55/attachment-0001.html>

From mailinglists at xgm.de  Thu Jan  3 09:57:29 2019
From: mailinglists at xgm.de (Florian Lindner)
Date: Thu, 3 Jan 2019 16:57:29 +0100
Subject: [petsc-users] Check vector for zeros
Message-ID: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>

Hello,

Happy New Year Everybody!

I get a vector from a linear solve and is used a divisor in VecPointwiseDivide. Clearly, I must check for zero entries before doing the division.

What is the best way to do so, especially performance wise?

The only way I come up with so far, is to VecCopy, VecAbs and then check for VecMin > eps. The VecCopy kind of scares me for the performance draw back. I haven't found something like a VecAbsMin.

Any better ideas?

Thanks,
Florian

From knepley at gmail.com  Thu Jan  3 10:12:03 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 11:12:03 -0500
Subject: [petsc-users] Check vector for zeros
In-Reply-To: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
References: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
Message-ID: <CAMYG4Gmu9SaCFCxZRNd4V07VTitOBwq56AuKfX8fth4dhsUSqg@mail.gmail.com>

On Thu, Jan 3, 2019 at 11:01 AM Florian Lindner via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> Happy New Year Everybody!
>
> I get a vector from a linear solve and is used a divisor in
> VecPointwiseDivide. Clearly, I must check for zero entries before doing the
> division.
>
> What is the best way to do so, especially performance wise?
>
> The only way I come up with so far, is to VecCopy, VecAbs and then check
> for VecMin > eps. The VecCopy kind of scares me for the performance draw
> back. I haven't found something like a VecAbsMin.
>
> Any better ideas?
>

I would write VecAbsMin() yourself,

https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-299
https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-1510

It should only be about 8 lines.

  Thanks,

    Matt


> Thanks,
> Florian
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/c4c3f918/attachment.html>

From stefano.zampini at gmail.com  Thu Jan  3 10:16:35 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 3 Jan 2019 17:16:35 +0100
Subject: [petsc-users] Check vector for zeros
In-Reply-To: <CAMYG4Gmu9SaCFCxZRNd4V07VTitOBwq56AuKfX8fth4dhsUSqg@mail.gmail.com>
References: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
	<CAMYG4Gmu9SaCFCxZRNd4V07VTitOBwq56AuKfX8fth4dhsUSqg@mail.gmail.com>
Message-ID: <01319048-F7DB-44B2-99B3-0766B57D45DE@gmail.com>

It already checks for zeros. See VecPointwiseDivide_Seq in src/vec/vec/impls/seq/bvec2.c

> On Jan 3, 2019, at 5:12 PM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Thu, Jan 3, 2019 at 11:01 AM Florian Lindner via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> Hello,
> 
> Happy New Year Everybody!
> 
> I get a vector from a linear solve and is used a divisor in VecPointwiseDivide. Clearly, I must check for zero entries before doing the division.
> 
> What is the best way to do so, especially performance wise?
> 
> The only way I come up with so far, is to VecCopy, VecAbs and then check for VecMin > eps. The VecCopy kind of scares me for the performance draw back. I haven't found something like a VecAbsMin.
> 
> Any better ideas?
> 
> I would write VecAbsMin() yourself,
> 
> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-299 <https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-299>
> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-1510 <https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-1510>
> 
> It should only be about 8 lines.
> 
>   Thanks,
> 
>     Matt
>  
> Thanks,
> Florian
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/a8d2e8ec/attachment.html>

From knepley at gmail.com  Thu Jan  3 10:18:27 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 11:18:27 -0500
Subject: [petsc-users] Check vector for zeros
In-Reply-To: <01319048-F7DB-44B2-99B3-0766B57D45DE@gmail.com>
References: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
	<CAMYG4Gmu9SaCFCxZRNd4V07VTitOBwq56AuKfX8fth4dhsUSqg@mail.gmail.com>
	<01319048-F7DB-44B2-99B3-0766B57D45DE@gmail.com>
Message-ID: <CAMYG4GnTjUKfjLH5sS45PZJMjxqjkm-UrNo6en2_UesKntO6+Q@mail.gmail.com>

On Thu, Jan 3, 2019 at 11:16 AM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> It already checks for zeros. See VecPointwiseDivide_Seq in
> src/vec/vec/impls/seq/bvec2.c
>

Yes, I was assuming he wanted to know if there were zeros and take some
other action.

  Thanks,

    Matt

> On Jan 3, 2019, at 5:12 PM, Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> On Thu, Jan 3, 2019 at 11:01 AM Florian Lindner via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello,
>>
>> Happy New Year Everybody!
>>
>> I get a vector from a linear solve and is used a divisor in
>> VecPointwiseDivide. Clearly, I must check for zero entries before doing the
>> division.
>>
>> What is the best way to do so, especially performance wise?
>>
>> The only way I come up with so far, is to VecCopy, VecAbs and then check
>> for VecMin > eps. The VecCopy kind of scares me for the performance draw
>> back. I haven't found something like a VecAbsMin.
>>
>> Any better ideas?
>>
>
> I would write VecAbsMin() yourself,
>
>
> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-299
>
> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/vec/vec/utils/vinv.c#lines-1510
>
> It should only be about 8 lines.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Florian
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/eb43cdef/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Thu Jan  3 11:23:29 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 3 Jan 2019 11:23:29 -0600
Subject: [petsc-users] Set diagonals other than main diagonal
Message-ID: <CAOGsD9i_7PK_ZqBh1RjH9=29vVM3jJ-Xw+cGGMmqtyJV_y-sQQ@mail.gmail.com>

Hi,

Is there any simple way of setting minor diagonals. The main diagonal can
be set by MatDiagonalSet but there's no equivalent way of doing it for for
minor diagonals. Does the preferred way to do this involve using
MatSetValues or there a simpler way?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/e7674176/attachment.html>

From knepley at gmail.com  Thu Jan  3 11:46:33 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 12:46:33 -0500
Subject: [petsc-users] Set diagonals other than main diagonal
In-Reply-To: <CAOGsD9i_7PK_ZqBh1RjH9=29vVM3jJ-Xw+cGGMmqtyJV_y-sQQ@mail.gmail.com>
References: <CAOGsD9i_7PK_ZqBh1RjH9=29vVM3jJ-Xw+cGGMmqtyJV_y-sQQ@mail.gmail.com>
Message-ID: <CAMYG4G=pbaTgx0DxReOjq=2T+TNRYjAo2STr5OPx7+REs8K_+A@mail.gmail.com>

On Thu, Jan 3, 2019 at 12:24 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> Is there any simple way of setting minor diagonals. The main diagonal can
> be set by MatDiagonalSet but there's no equivalent way of doing it for for
> minor diagonals. Does the preferred way to do this involve using
> MatSetValues or there a simpler way?
>

No, we do not have an interface for this. You really only see this
structure in 1D problems, so we have
not spent time on it.

  Thanks.

     Matt


> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/e1504fc8/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Jan  3 12:16:11 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 3 Jan 2019 18:16:11 +0000
Subject: [petsc-users] definition of level in PCILU
In-Reply-To: <CAE=bjn5OCENjEya-cUPgeBE5sjbpEqsOs7fg-kPH9=2k0YJGpQ@mail.gmail.com>
References: <CAE=bjn5OCENjEya-cUPgeBE5sjbpEqsOs7fg-kPH9=2k0YJGpQ@mail.gmail.com>
Message-ID: <7E6DF565-26BD-49EE-9DAE-F9D7B281C9FD@mcs.anl.gov>


   There is a hint in a comment in matimpl.h 

  Note: the level of factor(i,j) is set as lvl(i,j) = min{ lvl(i,j), lvl(i,prow)+lvl(prow,j)+1)

   So it uses the "sum" rule.

   Barry



> On Jan 3, 2019, at 9:38 AM, Tim Steinhoff via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear PETSc Team
> 
> I checked the references that are stated on
> 
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html
> 
> to get an idea of the definition of <level> that is used in PETSc's
> ILU(k) implementation. I haven't found anything of substance in the
> first reference (T. Dupont, R. Kendall, and H. Rachford) and only a
> rather vague description in the third one (Sec. 3.1 in the book). The
> second one isn't yet available to me.
> 
> Additionally, I checked
> 
> https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf
> 
> which gives a thorough explanation in Subsec. 10.3.3. Also, I took a look at
> 
> https://www.cs.odu.edu/~pothen/Papers/ilu-levels.pdf
> 
> which gives _two_ different definitions of level (p. 3 'sum' and
> 'max'). The sum-version equals the one in Saad. The statement on
> Wikipedia defines level according to the nonzero pattern of powers of
> A.
> 
> So the question is: Which definition of <level> was utilized for the
> implementation in PETSc? Any of the above? Or something else?
> 
> Thanks and kind regards
> tim
> 
> P.S.: Happy New Year!


From bsmith at mcs.anl.gov  Thu Jan  3 12:29:23 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 3 Jan 2019 18:29:23 +0000
Subject: [petsc-users] definition of level in PCILU
In-Reply-To: <CAE=bjn5OCENjEya-cUPgeBE5sjbpEqsOs7fg-kPH9=2k0YJGpQ@mail.gmail.com>
References: <CAE=bjn5OCENjEya-cUPgeBE5sjbpEqsOs7fg-kPH9=2k0YJGpQ@mail.gmail.com>
Message-ID: <84935657-7CC6-4AC5-833D-F3FB279F4175@anl.gov>


   There is a hint in a comment in matimpl.h 

  Note: the level of factor(i,j) is set as lvl(i,j) = min{ lvl(i,j), lvl(i,prow)+lvl(prow,j)+1)

   So it is the sum approach.

   Barry



> On Jan 3, 2019, at 9:38 AM, Tim Steinhoff via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear PETSc Team
> 
> I checked the references that are stated on
> 
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html
> 
> to get an idea of the definition of <level> that is used in PETSc's
> ILU(k) implementation. I haven't found anything of substance in the
> first reference (T. Dupont, R. Kendall, and H. Rachford) and only a
> rather vague description in the third one (Sec. 3.1 in the book). The
> second one isn't yet available to me.
> 
> Additionally, I checked
> 
> https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf
> 
> which gives a thorough explanation in Subsec. 10.3.3. Also, I took a look at
> 
> https://www.cs.odu.edu/~pothen/Papers/ilu-levels.pdf
> 
> which gives _two_ different definitions of level (p. 3 'sum' and
> 'max'). The sum-version equals the one in Saad. The statement on
> Wikipedia defines level according to the nonzero pattern of powers of
> A.
> 
> So the question is: Which definition of <level> was utilized for the
> implementation in PETSc? Any of the above? Or something else?
> 
> Thanks and kind regards
> tim
> 
> P.S.: Happy New Year!


From sajidsyed2021 at u.northwestern.edu  Thu Jan  3 14:21:30 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 3 Jan 2019 14:21:30 -0600
Subject: [petsc-users] Set diagonals other than main diagonal
In-Reply-To: <CAMYG4G=pbaTgx0DxReOjq=2T+TNRYjAo2STr5OPx7+REs8K_+A@mail.gmail.com>
References: <CAOGsD9i_7PK_ZqBh1RjH9=29vVM3jJ-Xw+cGGMmqtyJV_y-sQQ@mail.gmail.com>
	<CAMYG4G=pbaTgx0DxReOjq=2T+TNRYjAo2STr5OPx7+REs8K_+A@mail.gmail.com>
Message-ID: <CAOGsD9gLXw93YGtJhvNXvEi9aB4Z9Lj0JZnXnaraUYGqmg6o=g@mail.gmail.com>

Got it. Thank you!

On Thu, Jan 3, 2019 at 11:46 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 3, 2019 at 12:24 PM Sajid Ali via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> Is there any simple way of setting minor diagonals. The main diagonal can
>> be set by MatDiagonalSet but there's no equivalent way of doing it for for
>> minor diagonals. Does the preferred way to do this involve using
>> MatSetValues or there a simpler way?
>>
>
> No, we do not have an interface for this. You really only see this
> structure in 1D problems, so we have
> not spent time on it.
>
>   Thanks.
>
>      Matt
>
>
>> Thank You,
>> Sajid Ali
>> Applied Physics
>> Northwestern University
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/1c691d04/attachment.html>

From jed at jedbrown.org  Thu Jan  3 14:25:19 2019
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 03 Jan 2019 13:25:19 -0700
Subject: [petsc-users] Check vector for zeros
In-Reply-To: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
References: <d86bb5fa-558b-8fa7-3996-f75182ad1f45@xgm.de>
Message-ID: <87lg41ij4g.fsf@jedbrown.org>

Florian Lindner via petsc-users <petsc-users at mcs.anl.gov> writes:

> I get a vector from a linear solve and is used a divisor in VecPointwiseDivide. Clearly, I must check for zero entries before doing the division.

Note that VecPointwiseDivide does not divide by values that are
identically zero.

  for (i=0; i<n; i++) {
    if (yy[i] != 0.0) ww[i] = xx[i] / yy[i];
    else ww[i] = 0.0;
  }

This is semantically peculiar, but if it's okay for you, you can use
VecChop to truncate sufficiently small values to zero.

> What is the best way to do so, especially performance wise?
>
> The only way I come up with so far, is to VecCopy, VecAbs and then check for VecMin > eps. The VecCopy kind of scares me for the performance draw back. I haven't found something like a VecAbsMin.
>
> Any better ideas?
>
> Thanks,
> Florian

From danyang.su at gmail.com  Thu Jan  3 18:00:43 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Thu, 3 Jan 2019 16:00:43 -0800
Subject: [petsc-users] Installation error on macOS Mojave using GNU compiler
Message-ID: <E656C564-00E9-4869-B71F-536DFB4F431C@gmail.com>

Hi All,

I am trying to install PETSc on macOS Mojave using GNU compiler. 
First, I tried the debug version using the following configuration and it works fine.

./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-fblaslapack --download-hypre --download-superlu_dist --download-hdf5=yes --download-ctetgen

After testing debug version, I reconfigured PETSc with optimization turned on using the following configuration. However, I got error during this step.

./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-mpich --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-fblaslapack --download-hypre --download-superlu_dist --download-hdf5=yes --download-ctetgen --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native"

The error information is

ctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
/var/folders/jm/wcm4mv8s3v1gqz383tcf_4c00000gp/T//ccrlfFuo.s:14:2: error: instruction requires: AVX-512 ISA AVX-512 VL ISA
        vmovdqu64       (%rdi), %xmm0
        ^
make[2]: *** [src/binding/fortran/use_mpi/mpi_constants.mod-stamp] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2



Thanks,

Danyang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/5c5c8cd0/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 3296944 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/5c5c8cd0/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/5c5c8cd0/attachment-0003.html>

From sajidsyed2021 at u.northwestern.edu  Thu Jan  3 18:15:21 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 3 Jan 2019 18:15:21 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
Message-ID: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>

Hi,

I've compiled a program using petsc with debugging enabled.
Confirmation of the same : https://pastebin.com/aa0XDheD (check if the
loaded module is indeed petsc+debug, compile)

When I run it in a debugger, I get the error as shown below:
https://pastebin.com/GJtB2Ghz

What am I missing?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/b3ca0971/attachment.html>

From knepley at gmail.com  Thu Jan  3 18:45:03 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 19:45:03 -0500
Subject: [petsc-users] Installation error on macOS Mojave using GNU
 compiler
In-Reply-To: <E656C564-00E9-4869-B71F-536DFB4F431C@gmail.com>
References: <E656C564-00E9-4869-B71F-536DFB4F431C@gmail.com>
Message-ID: <CAMYG4GnxG3p8_BF_BcsAEE0xSkXVKJ6djSZSO7a5uiPh-AOHQQ@mail.gmail.com>

On Thu, Jan 3, 2019 at 7:02 PM Danyang Su via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi All,
>
> I am trying to install PETSc on macOS Mojave using GNU compiler.
> First, I tried the debug version using the following configuration and it
> works fine.
>
> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> --download-mpich --download-scalapack --download-parmetis --download-metis
> --download-ptscotch --download-fblaslapack --download-hypre
> --download-superlu_dist --download-hdf5=yes --download-ctetgen
>
> After testing debug version, I reconfigured PETSc with optimization turned
> on using the following configuration. However, I got error during this step.
>

Your optimization flags are not right because the compiler is producing
AVX-512, but your linker cannot handle it. However, it looks like
it might be that your Fortran compiler can't handle it. Do you need
Fortran? If not, turn it off (configure with --with-fc=0) and try again.

  Thanks,

    Matt


> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> --with-cxx-dialect=C++11 --download-mpich --download-scalapack
> --download-parmetis --download-metis --download-ptscotch
> --download-fblaslapack --download-hypre --download-superlu_dist
> --download-hdf5=yes --download-ctetgen --with-debugging=0 COPTFLAGS="-O3
> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
> FOPTFLAGS="-O3 -march=native -mtune=native"
>
> The error information is
>
> ctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
> file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
> file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> /var/folders/jm/wcm4mv8s3v1gqz383tcf_4c00000gp/T//ccrlfFuo.s:14:2: error:
> instruction requires: AVX-512 ISA AVX-512 VL ISA
>         vmovdqu64       (%rdi), %xmm0
>         ^
> make[2]: *** [src/binding/fortran/use_mpi/mpi_constants.mod-stamp] Error 1
> make[2]: *** Waiting for unfinished jobs....
> make[1]: *** [all-recursive] Error 1
> make: *** [all] Error 2
>
>
> Thanks,
>
> Danyang
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/a7fbdd1c/attachment.html>

From knepley at gmail.com  Thu Jan  3 18:48:14 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Jan 2019 19:48:14 -0500
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
Message-ID: <CAMYG4Gn7V=iGN9dpkjBD4cvuEQ=XjoXQiOHengKMsD=NJePC6Q@mail.gmail.com>

On Thu, Jan 3, 2019 at 7:16 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I've compiled a program using petsc with debugging enabled.
> Confirmation of the same : https://pastebin.com/aa0XDheD (check if the
> loaded module is indeed petsc+debug, compile)
>

I do not see -g on your compile line. Do you?

 Thanks,

    Matt


> When I run it in a debugger, I get the error as shown below:
> https://pastebin.com/GJtB2Ghz
>
> What am I missing?
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/86ce321e/attachment.html>

From balay at mcs.anl.gov  Thu Jan  3 18:59:21 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 4 Jan 2019 00:59:21 +0000
Subject: [petsc-users] Installation error on macOS Mojave using GNU
 compiler
In-Reply-To: <CAMYG4GnxG3p8_BF_BcsAEE0xSkXVKJ6djSZSO7a5uiPh-AOHQQ@mail.gmail.com>
References: <E656C564-00E9-4869-B71F-536DFB4F431C@gmail.com>
	<CAMYG4GnxG3p8_BF_BcsAEE0xSkXVKJ6djSZSO7a5uiPh-AOHQQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901031851460.20930@sb>

On Thu, 3 Jan 2019, Matthew Knepley via petsc-users wrote:

> On Thu, Jan 3, 2019 at 7:02 PM Danyang Su via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> 
> > Hi All,
> >
> > I am trying to install PETSc on macOS Mojave using GNU compiler.
> > First, I tried the debug version using the following configuration and it
> > works fine.
> >
> > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> > --download-mpich --download-scalapack --download-parmetis --download-metis
> > --download-ptscotch --download-fblaslapack --download-hypre
> > --download-superlu_dist --download-hdf5=yes --download-ctetgen
> >
> > After testing debug version, I reconfigured PETSc with optimization turned
> > on using the following configuration. However, I got error during this step.
> >
> 
> Your optimization flags are not right because the compiler is producing
> AVX-512, but your linker cannot handle it. However, it looks like
> it might be that your Fortran compiler can't handle it. Do you need
> Fortran? If not, turn it off (configure with --with-fc=0) and try again.

Or use 'FOPTFLAGS=-O3' [if its indeed fortran sources causing grief]

If '-march=native -mtune=native' gives compiler/linker errors - don't use them.

Satish

> 
>   Thanks,
> 
>     Matt
> 
> 
> > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> > --with-cxx-dialect=C++11 --download-mpich --download-scalapack
> > --download-parmetis --download-metis --download-ptscotch
> > --download-fblaslapack --download-hypre --download-superlu_dist
> > --download-hdf5=yes --download-ctetgen --with-debugging=0 COPTFLAGS="-O3
> > -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
> > FOPTFLAGS="-O3 -march=native -mtune=native"
> >
> > The error information is
> >
> > ctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
> > file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
> > file: .libs/libmpl.a(mpl_dbg.o) has no symbols
> > /var/folders/jm/wcm4mv8s3v1gqz383tcf_4c00000gp/T//ccrlfFuo.s:14:2: error:
> > instruction requires: AVX-512 ISA AVX-512 VL ISA
> >         vmovdqu64       (%rdi), %xmm0
> >         ^
> > make[2]: *** [src/binding/fortran/use_mpi/mpi_constants.mod-stamp] Error 1
> > make[2]: *** Waiting for unfinished jobs....
> > make[1]: *** [all-recursive] Error 1
> > make: *** [all] Error 2
> >
> >
> > Thanks,
> >
> > Danyang
> >
> 
> 
> 


From bsmith at mcs.anl.gov  Thu Jan  3 20:33:08 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 4 Jan 2019 02:33:08 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
Message-ID: <05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>


   This appears to be more a spack question than a PETSc question. That make (that doesn't have the -g) is controlled by spack, not PETSc.

   Barry


> On Jan 3, 2019, at 6:15 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi, 
> 
> I've compiled a program using petsc with debugging enabled. 
> Confirmation of the same : https://pastebin.com/aa0XDheD (check if the loaded module is indeed petsc+debug, compile)
> 
> When I run it in a debugger, I get the error as shown below: https://pastebin.com/GJtB2Ghz
> 
> What am I missing?
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University


From balay at mcs.anl.gov  Thu Jan  3 20:44:49 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 4 Jan 2019 02:44:49 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
Message-ID: <alpine.LFD.2.21.1901032038350.20930@sb>

I guess the primary bug here is 'petsc+debug' option is broken in spack.

Its not clear to me if a package in spack should have 'debug' option. [so I should remove this?]

since spack way is perhaps:

spack install CFLAGS=-g FFLAGS=-g CXXFLAGS=-g petsc

Satish

On Fri, 4 Jan 2019, Smith, Barry F. via petsc-users wrote:

> 
>    This appears to be more a spack question than a PETSc question. That make (that doesn't have the -g) is controlled by spack, not PETSc.
> 
>    Barry
> 
> 
> > On Jan 3, 2019, at 6:15 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hi, 
> > 
> > I've compiled a program using petsc with debugging enabled. 
> > Confirmation of the same : https://pastebin.com/aa0XDheD (check if the loaded module is indeed petsc+debug, compile)
> > 
> > When I run it in a debugger, I get the error as shown below: https://pastebin.com/GJtB2Ghz
> > 
> > What am I missing?
> > 
> > Thank You,
> > Sajid Ali
> > Applied Physics
> > Northwestern University
> 


From bsmith at mcs.anl.gov  Thu Jan  3 20:48:44 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 4 Jan 2019 02:48:44 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <alpine.LFD.2.21.1901032038350.20930@sb>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
Message-ID: <2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>



> On Jan 3, 2019, at 8:44 PM, Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> I guess the primary bug here is 'petsc+debug' option is broken in spack.
> 
> Its not clear to me if a package in spack should have 'debug' option. [so I should remove this?]

   Question  you should ask the spack folks. Many systems such as Xcode and Visual Studio have a debug and release mode, spack should also.

   Barry

> 
> since spack way is perhaps:
> 
> spack install CFLAGS=-g FFLAGS=-g CXXFLAGS=-g petsc
> 
> Satish
> 
> On Fri, 4 Jan 2019, Smith, Barry F. via petsc-users wrote:
> 
>> 
>>   This appears to be more a spack question than a PETSc question. That make (that doesn't have the -g) is controlled by spack, not PETSc.
>> 
>>   Barry
>> 
>> 
>>> On Jan 3, 2019, at 6:15 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi, 
>>> 
>>> I've compiled a program using petsc with debugging enabled. 
>>> Confirmation of the same : https://pastebin.com/aa0XDheD (check if the loaded module is indeed petsc+debug, compile)
>>> 
>>> When I run it in a debugger, I get the error as shown below: https://pastebin.com/GJtB2Ghz
>>> 
>>> What am I missing?
>>> 
>>> Thank You,
>>> Sajid Ali
>>> Applied Physics
>>> Northwestern University
>> 
> 


From yjwu16 at gmail.com  Fri Jan  4 08:47:12 2019
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Fri, 4 Jan 2019 09:47:12 -0500
Subject: [petsc-users] Problems about Picard and NolinearGS
In-Reply-To: <CAMYG4GkWUwgdX8PYAP-f4dqBgeKchq6b=bHRqpRyvF_uyqEWBg@mail.gmail.com>
References: <CAAMEhBjJKFAthSu11GKna7id6EHXTPHGuOhT-somy_=ST8_TAA@mail.gmail.com>
	<FFF11830-A669-496F-A750-EC009A463B84@anl.gov>
	<CAAMEhBhszto1xLSy7gqbM+ujjhWnKfdK-7wAQMRgsKhn_A9BJw@mail.gmail.com>
	<7BDB5EF2-6C9C-4AFD-A0AB-4C127596A174@mcs.anl.gov>
	<CAAMEhBjff8nrRJzo6n8W=8KkvL4THcRqWNqUPC3GBi7+vp3iBg@mail.gmail.com>
	<87ftuj2ef3.fsf@jedbrown.org>
	<CAAMEhBg9AViodmp3Oot2BJ7BXCxKmxTwYTfXTWDgM2zs0c+mdA@mail.gmail.com>
	<CAMYG4GkWUwgdX8PYAP-f4dqBgeKchq6b=bHRqpRyvF_uyqEWBg@mail.gmail.com>
Message-ID: <CAAMEhBgqptxzZKaZ2DNy-mQX-WWy3vGBi7O8Vuo-h-HN=n9eYA@mail.gmail.com>

Thank you very much for your answer.
I tried to use the quasi-Newton method to solve some examples, except
ex48(in src/snes/example/tutorial), the ex3 and the ex19 will produce the
same error information. The error information is as follows. Is there
anything I should pay attention to when using the quasi-Newton method?
args: -snes_type qn -snes_qn_restart_type periodic -snes_qn_scale_type
jacobian

In ex3, it has a routine about Jacobian, while in ex19 without a Jacobian
routine.
Is the quasi-Newton method use the inverse of the initial Jacobian matrix
by finite difference ?

Error information in ex3:
mpiexec -n 2 ./ex3 -snes_type qn -snes_qn_restart_type periodic
-snes_qn_scale_type jacobian -snes_view
atol=1e-50, rtol=1e-08, stol=1e-08, maxit=10000, maxf=30000
iter = 0,SNES Function norm 5.41468
iter = 1,SNES Function norm 0.806152
iter = 2,SNES Function norm 0.459274
iter = 3,SNES Function norm 0.0562937
iter = 4,SNES Function norm 0.00756409
iter = 5,SNES Function norm 0.00153315
iter = 6,SNES Function norm 0.000108099
iter = 7,SNES Function norm 2.77778e-06
iter = 8,SNES Function norm 5.56429e-08
iter = 9,SNES Function norm 5.56429e-08
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at beginning of function: Parameter number 2
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Fri Jan  4 09:11:00 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[0]PETSC ERROR: #1 VecValidValues() line 26 in
/home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Floating point exception
[1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at beginning of function: Parameter number 2
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Fri Jan  4 09:11:00 2019
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[1]PETSC ERROR: #1 VecValidValues() line 26 in
/home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
[1]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[1]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
[1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
[0]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
/home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
[0]PETSC ERROR: [1]PETSC ERROR: #5 SNESSolve_QN() line 403 in
/home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
[1]PETSC ERROR: #6 SNESSolve() line 4396 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
#6 SNESSolve() line 4396 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[0]PETSC ERROR: [1]PETSC ERROR: #7 main() line 277 in
/home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex3.c
#7 main() line 277 in
/home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex3.c
[0]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: [0]PETSC ERROR: -snes_qn_restart_type periodic
[0]PETSC ERROR: -snes_qn_restart_type periodic
[1]PETSC ERROR: -snes_qn_scale_type jacobian
[1]PETSC ERROR: -snes_qn_scale_type jacobian
[0]PETSC ERROR: -snes_type qn
[0]PETSC ERROR: -snes_type qn
[1]PETSC ERROR: -snes_view
-snes_view
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 72) - process 1

Error information in ex19
mpiexec -n 2 ./ex19 -snes_type qn -snes_qn_restart_type periodic
-snes_qn_scale_type jacobian -snes_view
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at beginning of function: Parameter number 2
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[0]PETSC ERROR: ./ex19 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Fri Jan  4 09:05:10 2019
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Floating point exception
[1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at beginning of function: Parameter number 2
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[1]PETSC ERROR: ./ex19 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Fri Jan  4 09:05:10 2019
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[1]PETSC ERROR: #1 VecValidValues() line 26 in
/home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
[1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack
[0]PETSC ERROR: #1 VecValidValues() line 26 in
/home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[0]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
[0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[1]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
[1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
[1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
/home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
/home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
[0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
/home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
[1]PETSC ERROR: #6 SNESSolve() line 4396 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[1]PETSC ERROR: #7 main() line 161 in
/home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex19.c
#6 SNESSolve() line 4396 in
/home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
[0]PETSC ERROR: #7 main() line 161 in
/home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -snes_qn_restart_type periodic
[0]PETSC ERROR: -snes_qn_restart_type periodic
[0]PETSC ERROR: -snes_qn_scale_type jacobian
[0]PETSC ERROR: -snes_type qn
[1]PETSC ERROR: -snes_qn_scale_type jacobian
[1]PETSC ERROR: -snes_type qn
[0]PETSC ERROR: -snes_view
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: -snes_view
[1]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 72) - process 1

I am very interested in quasi-Newton method, which may not be well
understood at the moment.
I look forward to your reply.

Thanks,
Yingjie


Matthew Knepley <knepley at gmail.com> ?2019?1?3??? ??8:36???

> On Thu, Jan 3, 2019 at 7:36 AM Yingjie Wu via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Thanks for your reply.
>> I read the article you provided. This is my first contact with the
>> quasi-Newton method.
>> I have some problem:
>> 1. From the point of view of algorithm, the quasi-Newton method does not
>> need to be solved by linear equations, so why would KSP be used?
>>
>
> It is solving the equations, but we know the analytical answer. KSP is our
> abstraction for linear equation solver,
> so we also use it in this case.
>
>
>> 2. Where and how to  use preconditioner in quasi-Newton method?
>>
>
> You do not need a PC here. Note that this is only going to work well if
> you have a good initial
> guess for the inverse of your Jacobian. The optimization people who
> invented have that (it is
> the identity). In Jed's paper, they use a V-cycle as the good initial
> guess.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Yingjie
>>
>> Jed Brown <jed at jedbrown.org> ?2018?12?27??? ??10:11???
>>
>>> Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov> writes:
>>>
>>> > I my opinion, the difficulty in constructing my Jacobian matrix is
>>> complex
>>> > coefficient.(eg, thermal conductivity* ?* , density )
>>> > For example, in temperature equation(T):
>>> >                 ?(*?*?T) - ?(?* Cp* u ) + Q =  0
>>> > *?* is thermal conductivity , ?* is density  Cp* is specific heat , u
>>> is
>>> > velocity, Q is source.
>>> > *? = *1.9*(1.0e-3)*pow(T+273.0-150.0,1.29)            function of T
>>> > ?=
>>> >
>>> (0.4814*P/1.0e3/(T+273.15))/(1.0+0.446*(1.0e-2)*P/1.0e3/(pow(T+273.15,1.2)))
>>> > function of T and P
>>> >
>>> > In theory, the coefficient contain variable. So it is complicated to
>>> > calculate the element of Jacobian.
>>> > In my snes_mf_operator method, I treat ?,? as constant. In every
>>> nonlinear
>>> > step, I use the solution update the ?,?  and thus update the
>>> > preconditioning matrix. At each residual function call(in
>>> > SNESFormFunction), I also update the coefficient to ensure the
>>> correction
>>> > of the residual function.
>>>
>>> If those Jacobian entries are really that hard to get right, you can try
>>> using quasi-Newton as an alternative to -snes_mf_operator; similar to
>>>
>>>
>>> https://jedbrown.org/files/BrownBrune-LowRankQuasiNewtonRobustJacobianLagging-2013.pdf
>>>
>>>
>>> In general, I would start with an exact Jacobian (computed by coloring,
>>> AD, or just in one specific material/configuration).  Test Newton using
>>> direct solvers to see your "best case" convergence (globalization may be
>>> needed).  Test fieldsplit using direct solvers on the blocks so you know
>>> how much error is attributable to that approximation.  Only then look at
>>> backing off on the approximation of the Jacobian and the preconditioner.
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/b76ba093/attachment-0001.html>

From yann.jobic at univ-amu.fr  Fri Jan  4 09:04:36 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Fri, 4 Jan 2019 16:04:36 +0100
Subject: [petsc-users] DMPlex H27 elements
Message-ID: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>

Dear Petsc Users,

I'm using DMPlexCreateFromCellList to create my DM. I would like to have 
an order 2 geometry.

It's working fine in 2D for elements of type Q9.

I checked that it's working correctly by using 
DMPlexComputeCellGeometryFEM, and compute the value of the determinant 
(CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)

I can import H8 elements, it's working fine.

But for H27 element, it's not working. I really don't know how to order 
my vertex in the cells array. So far, the determinant is zero...

I don't know where to look in order to find this information. I tried 
the function DMPlexGetRawFaces_Internal of the file

https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal

but it didn't help me.

Could you please point me where to look ?

Thanks, and happy new year !

Yann


From jed at jedbrown.org  Fri Jan  4 09:35:25 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 04 Jan 2019 08:35:25 -0700
Subject: [petsc-users] Problems about Picard and NolinearGS
In-Reply-To: <CAAMEhBgqptxzZKaZ2DNy-mQX-WWy3vGBi7O8Vuo-h-HN=n9eYA@mail.gmail.com>
References: <CAAMEhBjJKFAthSu11GKna7id6EHXTPHGuOhT-somy_=ST8_TAA@mail.gmail.com>
	<FFF11830-A669-496F-A750-EC009A463B84@anl.gov>
	<CAAMEhBhszto1xLSy7gqbM+ujjhWnKfdK-7wAQMRgsKhn_A9BJw@mail.gmail.com>
	<7BDB5EF2-6C9C-4AFD-A0AB-4C127596A174@mcs.anl.gov>
	<CAAMEhBjff8nrRJzo6n8W=8KkvL4THcRqWNqUPC3GBi7+vp3iBg@mail.gmail.com>
	<87ftuj2ef3.fsf@jedbrown.org>
	<CAAMEhBg9AViodmp3Oot2BJ7BXCxKmxTwYTfXTWDgM2zs0c+mdA@mail.gmail.com>
	<CAMYG4GkWUwgdX8PYAP-f4dqBgeKchq6b=bHRqpRyvF_uyqEWBg@mail.gmail.com>
	<CAAMEhBgqptxzZKaZ2DNy-mQX-WWy3vGBi7O8Vuo-h-HN=n9eYA@mail.gmail.com>
Message-ID: <87sgy8e8qq.fsf@jedbrown.org>

Yingjie Wu <yjwu16 at gmail.com> writes:

> Thank you very much for your answer.
> I tried to use the quasi-Newton method to solve some examples, except
> ex48(in src/snes/example/tutorial), the ex3 and the ex19 will produce the
> same error information. The error information is as follows. Is there
> anything I should pay attention to when using the quasi-Newton method?
> args: -snes_type qn -snes_qn_restart_type periodic -snes_qn_scale_type
> jacobian
>
> In ex3, it has a routine about Jacobian, while in ex19 without a Jacobian
> routine.
> Is the quasi-Newton method use the inverse of the initial Jacobian matrix
> by finite difference ?
>
> Error information in ex3:
> mpiexec -n 2 ./ex3 -snes_type qn -snes_qn_restart_type periodic
> -snes_qn_scale_type jacobian -snes_view
> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=10000, maxf=30000
> iter = 0,SNES Function norm 5.41468
> iter = 1,SNES Function norm 0.806152
> iter = 2,SNES Function norm 0.459274
> iter = 3,SNES Function norm 0.0562937
> iter = 4,SNES Function norm 0.00756409
> iter = 5,SNES Function norm 0.00153315
> iter = 6,SNES Function norm 0.000108099
> iter = 7,SNES Function norm 2.77778e-06
> iter = 8,SNES Function norm 5.56429e-08
> iter = 9,SNES Function norm 5.56429e-08
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception

-snes_qn_type broyden may help.  The default (BFGS) is for symmetric
problems and this formulation (with boundary conditions) is not
symmetric.

> [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
> at beginning of function: Parameter number 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Fri Jan  4 09:11:00 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [0]PETSC ERROR: #1 VecValidValues() line 26 in
> /home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Floating point exception
> [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
> at beginning of function: Parameter number 2
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Fri Jan  4 09:11:00 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [1]PETSC ERROR: #1 VecValidValues() line 26 in
> /home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [1]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
> [1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
> [0]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
> /home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
> [0]PETSC ERROR: [1]PETSC ERROR: #5 SNESSolve_QN() line 403 in
> /home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
> [1]PETSC ERROR: #6 SNESSolve() line 4396 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> #6 SNESSolve() line 4396 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [0]PETSC ERROR: [1]PETSC ERROR: #7 main() line 277 in
> /home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex3.c
> #7 main() line 277 in
> /home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex3.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: -snes_qn_restart_type periodic
> [0]PETSC ERROR: -snes_qn_restart_type periodic
> [1]PETSC ERROR: -snes_qn_scale_type jacobian
> [1]PETSC ERROR: -snes_qn_scale_type jacobian
> [0]PETSC ERROR: -snes_type qn
> [0]PETSC ERROR: -snes_type qn
> [1]PETSC ERROR: -snes_view
> -snes_view
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0
> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 1
>
> Error information in ex19
> mpiexec -n 2 ./ex19 -snes_type qn -snes_qn_restart_type periodic
> -snes_qn_scale_type jacobian -snes_view
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
> at beginning of function: Parameter number 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [0]PETSC ERROR: ./ex19 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Fri Jan  4 09:05:10 2019
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Floating point exception
> [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
> at beginning of function: Parameter number 2
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [1]PETSC ERROR: ./ex19 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Fri Jan  4 09:05:10 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [1]PETSC ERROR: #1 VecValidValues() line 26 in
> /home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
> [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack
> [0]PETSC ERROR: #1 VecValidValues() line 26 in
> /home/yjwu/petsc-3.10.1/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [0]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
> [0]PETSC ERROR: #2 SNESComputeFunction() line 2234 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [1]PETSC ERROR: #3 SNESLineSearchApply_CP() line 48 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/impls/cp/linesearchcp.c
> [1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
> [1]PETSC ERROR: #4 SNESLineSearchApply() line 648 in
> /home/yjwu/petsc-3.10.1/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
> /home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
> [0]PETSC ERROR: #5 SNESSolve_QN() line 403 in
> /home/yjwu/petsc-3.10.1/src/snes/impls/qn/qn.c
> [1]PETSC ERROR: #6 SNESSolve() line 4396 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [1]PETSC ERROR: #7 main() line 161 in
> /home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex19.c
> #6 SNESSolve() line 4396 in
> /home/yjwu/petsc-3.10.1/src/snes/interface/snes.c
> [0]PETSC ERROR: #7 main() line 161 in
> /home/yjwu/petsc-3.10.1/src/snes/examples/tutorials/ex19.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: -snes_qn_restart_type periodic
> [0]PETSC ERROR: -snes_qn_restart_type periodic
> [0]PETSC ERROR: -snes_qn_scale_type jacobian
> [0]PETSC ERROR: -snes_type qn
> [1]PETSC ERROR: -snes_qn_scale_type jacobian
> [1]PETSC ERROR: -snes_type qn
> [0]PETSC ERROR: -snes_view
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: -snes_view
> [1]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0
> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 1
>
> I am very interested in quasi-Newton method, which may not be well
> understood at the moment.
> I look forward to your reply.
>
> Thanks,
> Yingjie
>
>
> Matthew Knepley <knepley at gmail.com> ?2019?1?3??? ??8:36???
>
>> On Thu, Jan 3, 2019 at 7:36 AM Yingjie Wu via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Thanks for your reply.
>>> I read the article you provided. This is my first contact with the
>>> quasi-Newton method.
>>> I have some problem:
>>> 1. From the point of view of algorithm, the quasi-Newton method does not
>>> need to be solved by linear equations, so why would KSP be used?
>>>
>>
>> It is solving the equations, but we know the analytical answer. KSP is our
>> abstraction for linear equation solver,
>> so we also use it in this case.
>>
>>
>>> 2. Where and how to  use preconditioner in quasi-Newton method?
>>>
>>
>> You do not need a PC here. Note that this is only going to work well if
>> you have a good initial
>> guess for the inverse of your Jacobian. The optimization people who
>> invented have that (it is
>> the identity). In Jed's paper, they use a V-cycle as the good initial
>> guess.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>> Yingjie
>>>
>>> Jed Brown <jed at jedbrown.org> ?2018?12?27??? ??10:11???
>>>
>>>> Yingjie Wu via petsc-users <petsc-users at mcs.anl.gov> writes:
>>>>
>>>> > I my opinion, the difficulty in constructing my Jacobian matrix is
>>>> complex
>>>> > coefficient.(eg, thermal conductivity* ?* , density )
>>>> > For example, in temperature equation(T):
>>>> >                 ?(*?*?T) - ?(?* Cp* u ) + Q =  0
>>>> > *?* is thermal conductivity , ?* is density  Cp* is specific heat , u
>>>> is
>>>> > velocity, Q is source.
>>>> > *? = *1.9*(1.0e-3)*pow(T+273.0-150.0,1.29)            function of T
>>>> > ?=
>>>> >
>>>> (0.4814*P/1.0e3/(T+273.15))/(1.0+0.446*(1.0e-2)*P/1.0e3/(pow(T+273.15,1.2)))
>>>> > function of T and P
>>>> >
>>>> > In theory, the coefficient contain variable. So it is complicated to
>>>> > calculate the element of Jacobian.
>>>> > In my snes_mf_operator method, I treat ?,? as constant. In every
>>>> nonlinear
>>>> > step, I use the solution update the ?,?  and thus update the
>>>> > preconditioning matrix. At each residual function call(in
>>>> > SNESFormFunction), I also update the coefficient to ensure the
>>>> correction
>>>> > of the residual function.
>>>>
>>>> If those Jacobian entries are really that hard to get right, you can try
>>>> using quasi-Newton as an alternative to -snes_mf_operator; similar to
>>>>
>>>>
>>>> https://jedbrown.org/files/BrownBrune-LowRankQuasiNewtonRobustJacobianLagging-2013.pdf
>>>>
>>>>
>>>> In general, I would start with an exact Jacobian (computed by coloring,
>>>> AD, or just in one specific material/configuration).  Test Newton using
>>>> direct solvers to see your "best case" convergence (globalization may be
>>>> needed).  Test fieldsplit using direct solvers on the blocks so you know
>>>> how much error is attributable to that approximation.  Only then look at
>>>> backing off on the approximation of the Jacobian and the preconditioner.
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>

From sajidsyed2021 at u.northwestern.edu  Fri Jan  4 10:24:51 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 4 Jan 2019 10:24:51 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
Message-ID: <CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>

Spack changes the configuration if debug variant is enabled as per this
line :

'--with-debugging=%s' % ('1' if '+debug' in spec else '0'),

I'm not sure what needs to be done to change this.

In the meantime, I added -g to CFLAGS in the makefile and now I have this :
https://pastebin.com/wxf05BH0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/ea80ed10/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Fri Jan  4 10:30:33 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 4 Jan 2019 10:30:33 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
Message-ID: <CAOGsD9jiNj4SA=LV_xhqDkjZ6aYiR398Z-q=xXrdzwszeQ61uA@mail.gmail.com>

Should gdb : https://spack.readthedocs.io/en/latest/package_list.html#gdb
be listed as an explicit dependency ?

Also, why is the error message asking me to re-install my system
blas/lapack ? I installed petsc with intel-mkl as blas/lapack provider and
the linking failed for superlu-dist. Is that carrying over here ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/2fbf3e93/attachment.html>

From danyang.su at gmail.com  Fri Jan  4 11:10:12 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Fri, 04 Jan 2019 09:10:12 -0800
Subject: [petsc-users] Installation error on macOS Mojave using GNU
 compiler
In-Reply-To: <alpine.LFD.2.21.1901031851460.20930@sb>
References: <E656C564-00E9-4869-B71F-536DFB4F431C@gmail.com>
	<CAMYG4GnxG3p8_BF_BcsAEE0xSkXVKJ6djSZSO7a5uiPh-AOHQQ@mail.gmail.com>
	<alpine.LFD.2.21.1901031851460.20930@sb>
Message-ID: <3692DEBF-5F9A-44C0-8722-8BCE98792D10@gmail.com>



?On 2019-01-03, 4:59 PM, "Balay, Satish" <balay at mcs.anl.gov> wrote:

    On Thu, 3 Jan 2019, Matthew Knepley via petsc-users wrote:
    
    > On Thu, Jan 3, 2019 at 7:02 PM Danyang Su via petsc-users <
    > petsc-users at mcs.anl.gov> wrote:
    > 
    > > Hi All,
    > >
    > > I am trying to install PETSc on macOS Mojave using GNU compiler.
    > > First, I tried the debug version using the following configuration and it
    > > works fine.
    > >
    > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
    > > --download-mpich --download-scalapack --download-parmetis --download-metis
    > > --download-ptscotch --download-fblaslapack --download-hypre
    > > --download-superlu_dist --download-hdf5=yes --download-ctetgen
    > >
    > > After testing debug version, I reconfigured PETSc with optimization turned
    > > on using the following configuration. However, I got error during this step.
    > >
    > 
    > Your optimization flags are not right because the compiler is producing
    > AVX-512, but your linker cannot handle it. However, it looks like
    > it might be that your Fortran compiler can't handle it. Do you need
    > Fortran? If not, turn it off (configure with --with-fc=0) and try again.
    
    Or use 'FOPTFLAGS=-O3' [if its indeed fortran sources causing grief]
    
    If '-march=native -mtune=native' gives compiler/linker errors - don't use them.
    
    Satish

Hi Satish,

After removing "-march=native -mtune=native", it works now.

Thanks,

Danyang
    
    > 
    >   Thanks,
    > 
    >     Matt
    > 
    > 
    > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
    > > --with-cxx-dialect=C++11 --download-mpich --download-scalapack
    > > --download-parmetis --download-metis --download-ptscotch
    > > --download-fblaslapack --download-hypre --download-superlu_dist
    > > --download-hdf5=yes --download-ctetgen --with-debugging=0 COPTFLAGS="-O3
    > > -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
    > > FOPTFLAGS="-O3 -march=native -mtune=native"
    > >
    > > The error information is
    > >
    > > ctoolchain/usr/bin/ranlib: file: .libs/libmpl.a(mpl_dbg.o) has no symbols
    > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
    > > file: .libs/libmpl.a(mpl_dbg.o) has no symbols
    > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
    > > file: .libs/libmpl.a(mpl_dbg.o) has no symbols
    > > /var/folders/jm/wcm4mv8s3v1gqz383tcf_4c00000gp/T//ccrlfFuo.s:14:2: error:
    > > instruction requires: AVX-512 ISA AVX-512 VL ISA
    > >         vmovdqu64       (%rdi), %xmm0
    > >         ^
    > > make[2]: *** [src/binding/fortran/use_mpi/mpi_constants.mod-stamp] Error 1
    > > make[2]: *** Waiting for unfinished jobs....
    > > make[1]: *** [all-recursive] Error 1
    > > make: *** [all] Error 2
    > >
    > >
    > > Thanks,
    > >
    > > Danyang
    > >
    > 
    > 
    > 
    
    



From sajidsyed2021 at u.northwestern.edu  Fri Jan  4 11:57:24 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 4 Jan 2019 11:57:24 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9jiNj4SA=LV_xhqDkjZ6aYiR398Z-q=xXrdzwszeQ61uA@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<CAOGsD9jiNj4SA=LV_xhqDkjZ6aYiR398Z-q=xXrdzwszeQ61uA@mail.gmail.com>
Message-ID: <CAOGsD9j5bDo+jPJvxKauJCg6g6DvOuvCBEOVnFECe+E3yY3Ntg@mail.gmail.com>

I asked in the spack-slack channel and was told to include the explicitly
link the gfortran associated with the gcc version I build petsc with.
Following that advice, I'm using gcc at 7.3.0, linking gcc at 7.3.0/lib64 in
FFLAGS and setting -g option to CFLAGS in my makefile.

Yet, somehow the executable still picks up the system gfortran (It picks
both libgfortran.so.4 and libgfortran.so.3). Anything else I missed trying?

PS: My makefile looks like this:

[sajid at xrm free_space]$ cat makefile

ALL:

CFLAGS          = -g
FFLAGS          =
-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/
CPPFLAGS        =
FPPFLAGS        =
CLEANFILES      = ex_matlab

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

# your targets if any

include ${PETSC_DIR}/lib/petsc/conf/test

I've looked into the superludist and can confirm that it is indeed linked
to the system gfortran due to it's build error and since it's a petsc
dependency, it might be that this is where the system gfortran is coming in
instead of the newer gfortran associated with gcc-7.3.0 but I'm not sure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/da959a42/attachment.html>

From bsmith at mcs.anl.gov  Fri Jan  4 11:59:19 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 4 Jan 2019 17:59:19 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
Message-ID: <9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>


  Why did you quit the debugger?

(gdb) quit  

Why didn't set the appropriate gdb option for catching floating point exceptions and then type cont ?

  Barry


> On Jan 4, 2019, at 10:24 AM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> Spack changes the configuration if debug variant is enabled as per this line :
> 
> '--with-debugging=%s' % ('1' if '+debug' in spec else '0'), 
> 
> I'm not sure what needs to be done to change this. 
> 
> In the meantime, I added -g to CFLAGS in the makefile and now I have this : https://pastebin.com/wxf05BH0


From sajidsyed2021 at u.northwestern.edu  Fri Jan  4 14:50:34 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 4 Jan 2019 14:50:34 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
Message-ID: <CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>

Going by the advice given here : https://stackoverflow.com/a/2792735

I tried running the program, hoping to catch the FP exception and I got the
error I had before (which turned out to be due to an FP exception as per
previous petsc thread):

Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64
bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64
elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7.x86_64
lapack-3.4.2-8.el7.x86_64 libattr-2.4.46-12.el7.x86_64
libcap-2.22-9.el7.x86_64 libgfortran-4.8.5-16.el7.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 systemd-libs-219-42.el7_4.4.x86_64
xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) run -ts_type cn
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Killed
[sajid at xrm free_space]$ Starting program:
/raid/home/sajid/packages/xwp_petsc/2d/free_space/./ex_modify -ts_type cn
[setting tty state failed in terminal_inferior: Input/output error]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File
"/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.
so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load
safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
Solving a linear TS problem on 1 processor
mx : 512, my: 512 lambda : 1.239840e-10
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR:
[0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase
-ts_max_snes_failures or make negative to attempt recovery
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision:
bd27d3f284687498e4c4678d234c0e308a5bc236  GIT Date: 2019-01-02 11:00:20
-0600
[0]PETSC ERROR:
/raid/home/sajid/packages/xwp_petsc/2d/free_space/./ex_modify on a  named
xrm by sajid Fri Jan  4 14:47:56 2019
[0]PETSC ERROR: Configure options
--prefix=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj
--with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0
CFLAGS= FFLAGS= CXXFLAGS= COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS=
--with-cc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc
--with-cxx=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpic++
--with-fc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpif90
--with-precision=double --with-scalar-type=complex
--with-shared-libraries=1 --with-debugging=1 --with-64-bit-indices=0
--with-blaslapack-lib="/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_intel_lp64.so
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_sequential.so
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_core.so
/lib64/libpthread.so /lib64/libm.so /lib64/libdl.so" --with-x=1
--with-clanguage=C --with-scalapack=0 --with-metis=1
--with-metis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/metis-5.1.0-nhgzn4kjskctzmzv35mstvd34nj2ugek
--with-hdf5=1
--with-hdf5-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww
--with-hypre=0 --with-parmetis=1
--with-parmetis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/parmetis-4.0.3-hw3j2ss7mjsc5x5f2gaflirnuufzptil
--with-mumps=0 --with-trilinos=0 --with-cxx-dialect=C++11
--with-superlu_dist-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/include
--with-superlu_dist-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/lib/libsuperlu_dist.a
--with-superlu_dist=1 --with-suitesparse=0
--with-zlib-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/include
--with-zlib-lib="-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/lib
-lz" --with-zlib=1
[0]PETSC ERROR: #1 TSStep() line 3673 in
/tmp/sajid/spack-stage/spack-stage-0boEG4/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #2 TSSolve() line 3847 in
/tmp/sajid/spack-stage/spack-stage-0boEG4/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #3 main() line 183 in
/raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -ts_type cn
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 91) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=91
:
system msg for write_line failure : Bad file descriptor
[Inferior 1 (process 34440) exited with code 0133]
(gdb)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/72cc82f5/attachment.html>

From mattalo.k at gmail.com  Fri Jan  4 15:23:44 2019
From: mattalo.k at gmail.com (Kevin Mattalo)
Date: Fri, 4 Jan 2019 16:23:44 -0500
Subject: [petsc-users] Using PETSc Data Types with Existing F90 Functions
Message-ID: <CAPiecX0esAvpg-BBCHu-x0W3dQV2zVZWe4-4N1af-AHv9=v99Q@mail.gmail.com>

Hello,

I am currently implementing an implicit discretization of a high-order
DG method that uses a pseudo-transient continuation globalization
procedure combined with an inexact Newton method. I am using GMRES for
the linear solves with a preconditioner constructed from the exact
analytic Jacobian.

In order to verify my Jacobian (mostly the linearization of the flux
functions since I have already verified the linearization of the local
volume terms) I was trying to implement the finite difference
approximation (SNESComputeJacobianDefault()). However, since all of my
functions for calculating the residual use intrinsic F90 arrays for
the MPI and the actual computations, I was wondering if there is a way
to extract the values from the PETSc data types (Vec, Mat etc.) and to
input the extracted values into the F90 arrays to use in my F90
functions. The reason I need to do this is because I am assuming the
ComputeJacobianDefault() function perturbs the argument x (the current
point) which is a Vec datatype. Therefore, in order to find the
residual using my existing functions (which don't use PETSc data
types) I have to copy the values in x into my existing F90 arrays in
order to use my existing functions. I tried simply equating the
vectors but I believe that only copied the address and not the actual
values within the vector.

Thanks,

Kevin

From sajidsyed2021 at u.northwestern.edu  Fri Jan  4 15:24:16 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 4 Jan 2019 15:24:16 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
Message-ID: <CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>

Trying it slightly differently, I do see it's a SIGFPE, arithmetic
exception but all it shows is that it's an error in TSSolve but no further
than that.

[sajid at xrm free_space]$ gdb ex_modify
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify...done.
(gdb) run -ts_type cn --args -fp_trap
Starting program:
/raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify -ts_type cn
--args -fp_trap
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File
"/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.
so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load
safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
        add-auto-load-safe-path
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.
so.6.0.24-gdb.py
line to your configuration file "/raid/home/sajid/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/raid/home/sajid/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the
shell:
        info "(gdb)Auto-loading safe path"
Solving a linear TS problem on 1 processor
mx : 512, my: 512 lambda : 1.239840e-10

Program received signal SIGFPE, Arithmetic exception.
__muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305,
c=-0.0024992568840190117, d=0.024886737403015963)
    at
/raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
1978
/raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:
No such file or directory.
Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64
bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64
elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7.x86_64
lapack-3.4.2-8.el7.x86_64 libattr-2.4.46-12.el7.x86_64
libcap-2.22-9.el7.x86_64 libgfortran-4.8.5-16.el7.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 systemd-libs-219-42.el7_4.4.x86_64
xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305,
c=-0.0024992568840190117, d=0.024886737403015963)
    at
/raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
#1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#2  0x00007ffff5ba61a8 in MatSolve ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#3  0x00007ffff6bc8a55 in PCApply_ILU ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#4  0x00007ffff6cde6eb in PCApply ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#5  0x00007ffff6e3ad4a in KSP_PCApply ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#6  0x00007ffff6e3bc36 in KSPInitialResidual ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#7  0x00007ffff6dc0736 in KSPSolve_GMRES ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#8  0x00007ffff6e1158e in KSPSolve ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#9  0x00007ffff6fac311 in SNESSolve_KSPONLY ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#10 0x00007ffff6f346c7 in SNESSolve ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#11 0x00007ffff714d58b in TSTheta_SNESSolve ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#12 0x00007ffff714df63 in TSStep_Theta ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#13 0x00007ffff7053ebe in TSStep ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
---Type <return> to continue, or q <return> to quit---
#14 0x00007ffff70560bc in TSSolve ()
   from
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
#15 0x000000000040271d in main (argc=5, argv=0x7fffffffce38) at
/raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify.c:183
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/fc66ccfe/attachment.html>

From bsmith at mcs.anl.gov  Fri Jan  4 18:56:26 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 5 Jan 2019 00:56:26 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
Message-ID: <302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>

 
   Looks like spack did not compile PETSc with the -g option; which is strange because in both debug and and --with-debugging=0 PETSc ./configure make uses debug symbols

#0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
    at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
#1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
   from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010

   Anyway the FPE occurs in MatSolve_SeqAIJ_NaturalOrdering which usually indicates a zero on the diagonal of the matrix. Is that possible?

   Barry


    Satish,

      Does spack somehow turn off the -g ?

> On Jan 4, 2019, at 3:24 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> Trying it slightly differently, I do see it's a SIGFPE, arithmetic exception but all it shows is that it's an error in TSSolve but no further than that. 
> 
> [sajid at xrm free_space]$ gdb ex_modify 
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify...done.
> (gdb) run -ts_type cn --args -fp_trap
> Starting program: /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify -ts_type cn --args -fp_trap
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> warning: File "/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
> To enable execution of this file add
>         add-auto-load-safe-path /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.so.6.0.24-gdb.py
> line to your configuration file "/raid/home/sajid/.gdbinit".
> To completely disable this security protection add
>         set auto-load safe-path /
> line to your configuration file "/raid/home/sajid/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
>         info "(gdb)Auto-loading safe path"
> Solving a linear TS problem on 1 processor
> mx : 512, my: 512 lambda : 1.239840e-10
> 
> Program received signal SIGFPE, Arithmetic exception.
> __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
>     at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> 1978    /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c: No such file or directory.
> Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7.x86_64 lapack-3.4.2-8.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libgfortran-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 systemd-libs-219-42.el7_4.4.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
>     at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> #1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #2  0x00007ffff5ba61a8 in MatSolve ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #3  0x00007ffff6bc8a55 in PCApply_ILU ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #4  0x00007ffff6cde6eb in PCApply ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #5  0x00007ffff6e3ad4a in KSP_PCApply ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #6  0x00007ffff6e3bc36 in KSPInitialResidual ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #7  0x00007ffff6dc0736 in KSPSolve_GMRES ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #8  0x00007ffff6e1158e in KSPSolve ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #9  0x00007ffff6fac311 in SNESSolve_KSPONLY ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #10 0x00007ffff6f346c7 in SNESSolve ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #11 0x00007ffff714d58b in TSTheta_SNESSolve ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #12 0x00007ffff714df63 in TSStep_Theta ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #13 0x00007ffff7053ebe in TSStep ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> ---Type <return> to continue, or q <return> to quit---
> #14 0x00007ffff70560bc in TSSolve ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> #15 0x000000000040271d in main (argc=5, argv=0x7fffffffce38) at /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify.c:183
> 


From bsmith at mcs.anl.gov  Fri Jan  4 18:59:32 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 5 Jan 2019 00:59:32 +0000
Subject: [petsc-users] Using PETSc Data Types with Existing F90 Functions
In-Reply-To: <CAPiecX0esAvpg-BBCHu-x0W3dQV2zVZWe4-4N1af-AHv9=v99Q@mail.gmail.com>
References: <CAPiecX0esAvpg-BBCHu-x0W3dQV2zVZWe4-4N1af-AHv9=v99Q@mail.gmail.com>
Message-ID: <EC479321-C3C7-434F-9012-D2B45746D3F7@anl.gov>


   VecGetArrayF90()? Then a loop over the entries?

> On Jan 4, 2019, at 3:23 PM, Kevin Mattalo via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
> 
> I am currently implementing an implicit discretization of a high-order
> DG method that uses a pseudo-transient continuation globalization
> procedure combined with an inexact Newton method. I am using GMRES for
> the linear solves with a preconditioner constructed from the exact
> analytic Jacobian.
> 
> In order to verify my Jacobian (mostly the linearization of the flux
> functions since I have already verified the linearization of the local
> volume terms) I was trying to implement the finite difference
> approximation (SNESComputeJacobianDefault()). However, since all of my
> functions for calculating the residual use intrinsic F90 arrays for
> the MPI and the actual computations, I was wondering if there is a way
> to extract the values from the PETSc data types (Vec, Mat etc.) and to
> input the extracted values into the F90 arrays to use in my F90
> functions. The reason I need to do this is because I am assuming the
> ComputeJacobianDefault() function perturbs the argument x (the current
> point) which is a Vec datatype. Therefore, in order to find the
> residual using my existing functions (which don't use PETSc data
> types) I have to copy the values in x into my existing F90 arrays in
> order to use my existing functions. I tried simply equating the
> vectors but I believe that only copied the address and not the actual
> values within the vector.
> 
> Thanks,
> 
> Kevin


From knepley at gmail.com  Fri Jan  4 19:29:48 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Jan 2019 20:29:48 -0500
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
Message-ID: <CAMYG4GnvJcLWoVC+M0G74dBdOM8jUmrDXD2EX6ZiL13HJUJoXg@mail.gmail.com>

On Fri, Jan 4, 2019 at 7:56 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>    Looks like spack did not compile PETSc with the -g option; which is
> strange because in both debug and and --with-debugging=0 PETSc ./configure
> make uses debug symbols
>
> #0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305,
> c=-0.0024992568840190117, d=0.024886737403015963)
>     at
> /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> #1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
>    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
>
>    Anyway the FPE occurs in MatSolve_SeqAIJ_NaturalOrdering which usually
> indicates a zero on the diagonal of the matrix. Is that possible?
>
>    Barry
>
>
>     Satish,
>
>       Does spack somehow turn off the -g ?
>

I bet spack overrides the CFLAGS, which has the -g, instead of using
COPTFLAGS like we advise.

The amount of time spent working around spack FAR OUTWEIGHS any possible
benefit it might have. Just build the library, It takes 5 min.

   Matt


> > On Jan 4, 2019, at 3:24 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu>
> wrote:
> >
> > Trying it slightly differently, I do see it's a SIGFPE, arithmetic
> exception but all it shows is that it's an error in TSSolve but no further
> than that.
> >
> > [sajid at xrm free_space]$ gdb ex_modify
> > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show
> copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from
> /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify...done.
> > (gdb) run -ts_type cn --args -fp_trap
> > Starting program:
> /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify -ts_type cn
> --args -fp_trap
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > warning: File
> "/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.
> so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load
> safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
> > To enable execution of this file add
> >         add-auto-load-safe-path
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.
> so.6.0.24-gdb.py
> > line to your configuration file "/raid/home/sajid/.gdbinit".
> > To completely disable this security protection add
> >         set auto-load safe-path /
> > line to your configuration file "/raid/home/sajid/.gdbinit".
> > For more information about this security protection see the
> > "Auto-loading safe path" section in the GDB manual.  E.g., run from the
> shell:
> >         info "(gdb)Auto-loading safe path"
> > Solving a linear TS problem on 1 processor
> > mx : 512, my: 512 lambda : 1.239840e-10
> >
> > Program received signal SIGFPE, Arithmetic exception.
> > __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305,
> c=-0.0024992568840190117, d=0.024886737403015963)
> >     at
> /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> > 1978
> /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:
> No such file or directory.
> > Missing separate debuginfos, use: debuginfo-install
> blas-3.4.2-8.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64
> elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64
> glibc-2.17-196.el7.x86_64 lapack-3.4.2-8.el7.x86_64
> libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64
> libgfortran-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
> systemd-libs-219-42.el7_4.4.x86_64 xz-libs-5.2.2-1.el7.x86_64
> zlib-1.2.7-17.el7.x86_64
> > (gdb) bt
> > #0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305,
> c=-0.0024992568840190117, d=0.024886737403015963)
> >     at
> /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> > #1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #2  0x00007ffff5ba61a8 in MatSolve ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #3  0x00007ffff6bc8a55 in PCApply_ILU ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #4  0x00007ffff6cde6eb in PCApply ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #5  0x00007ffff6e3ad4a in KSP_PCApply ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #6  0x00007ffff6e3bc36 in KSPInitialResidual ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #7  0x00007ffff6dc0736 in KSPSolve_GMRES ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #8  0x00007ffff6e1158e in KSPSolve ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #9  0x00007ffff6fac311 in SNESSolve_KSPONLY ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #10 0x00007ffff6f346c7 in SNESSolve ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #11 0x00007ffff714d58b in TSTheta_SNESSolve ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #12 0x00007ffff714df63 in TSStep_Theta ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #13 0x00007ffff7053ebe in TSStep ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > ---Type <return> to continue, or q <return> to quit---
> > #14 0x00007ffff70560bc in TSSolve ()
> >    from
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #15 0x000000000040271d in main (argc=5, argv=0x7fffffffce38) at
> /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify.c:183
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/43810629/attachment.html>

From knepley at gmail.com  Fri Jan  4 19:36:14 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Jan 2019 20:36:14 -0500
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
Message-ID: <CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>

On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear Petsc Users,
>
> I'm using DMPlexCreateFromCellList to create my DM. I would like to have
> an order 2 geometry.
>
> It's working fine in 2D for elements of type Q9.
>

I do not see how that is possible. The topology does not work that way, and
there
is no way a 9 vertex quad was interpolated, so something else is happening.

Moreover, anything you input still has affine geometry. Toby has been
working on
non-affine geometry, and all the code is there, but right now you have to
build and
populate the coordinate space by hand. What this means is that you create a
quadratic space for the coordinate DM (rather than the default linear), get
the
global coordinate vector out, and stick in the Q9 coordinates. The topology
is still
the same no matter what coordinates you read in.

We will get around to making an interface for this soon.

  Thanks,

     Matt


> I checked that it's working correctly by using
> DMPlexComputeCellGeometryFEM, and compute the value of the determinant
> (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>
> I can import H8 elements, it's working fine.
>
> But for H27 element, it's not working. I really don't know how to order
> my vertex in the cells array. So far, the determinant is zero...
>
> I don't know where to look in order to find this information. I tried
> the function DMPlexGetRawFaces_Internal of the file
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>
> but it didn't help me.
>
> Could you please point me where to look ?
>
> Thanks, and happy new year !
>
> Yann
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190104/66c8c170/attachment.html>

From balay at mcs.anl.gov  Fri Jan  4 19:46:37 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sat, 5 Jan 2019 01:46:37 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
Message-ID: <alpine.LFD.2.21.1901041940270.20930@sb>

I had made a change so that configure uses CFLAGS etc from spack - so diabled internal COPTFLAGS guess within petsc.

I haven't figure out the correct way to do debug vs release builds in spack yet.

So the +debug option is broken [as mentioned before].

One workarround is is to improve '+debug' option for now with :

diff --git a/var/spack/repos/builtin/packages/petsc/package.py b/var/spack/repos/builtin/packages/petsc/package.py
index 99800b501..3fba7cf72 100644
--- a/var/spack/repos/builtin/packages/petsc/package.py
+++ b/var/spack/repos/builtin/packages/petsc/package.py
@@ -195,10 +195,7 @@ class Petsc(Package):
                    '--download-hwloc=0',
                    'CFLAGS=%s' % ' '.join(spec.compiler_flags['cflags']),
                    'FFLAGS=%s' % ' '.join(spec.compiler_flags['fflags']),
-                   'CXXFLAGS=%s' % ' '.join(spec.compiler_flags['cxxflags']),
-                   'COPTFLAGS=',
-                   'FOPTFLAGS=',
-                   'CXXOPTFLAGS=']
+                   'CXXFLAGS=%s' % ' '.join(spec.compiler_flags['cxxflags'])]
         options.extend(self.mpi_dependent_options())
         options.extend([
             '--with-precision=%s' % (
@@ -209,6 +206,13 @@ class Petsc(Package):
             '--with-debugging=%s' % ('1' if '+debug' in spec else '0'),
             '--with-64-bit-indices=%s' % ('1' if '+int64' in spec else '0')
         ])
+        if '+debug' in spec:
+            options.append('--with-debugging=%s')
+        else:
+            options.extend('COPTFLAGS=',
+                           'FOPTFLAGS=',
+                           'CXXOPTFLAGS=')
+
         # Make sure we use exactly the same Blas/Lapack libraries
         # across the DAG. To that end list them explicitly
         lapack_blas = spec['lapack'].libs + spec['blas'].libs

Satish

On Sat, 5 Jan 2019, Smith, Barry F. via petsc-users wrote:

>  
>    Looks like spack did not compile PETSc with the -g option; which is strange because in both debug and and --with-debugging=0 PETSc ./configure make uses debug symbols
> 
> #0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
>     at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> #1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
>    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> 
>    Anyway the FPE occurs in MatSolve_SeqAIJ_NaturalOrdering which usually indicates a zero on the diagonal of the matrix. Is that possible?
> 
>    Barry
> 
> 
>     Satish,
> 
>       Does spack somehow turn off the -g ?
> 
> > On Jan 4, 2019, at 3:24 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> > 
> > Trying it slightly differently, I do see it's a SIGFPE, arithmetic exception but all it shows is that it's an error in TSSolve but no further than that. 
> > 
> > [sajid at xrm free_space]$ gdb ex_modify 
> > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify...done.
> > (gdb) run -ts_type cn --args -fp_trap
> > Starting program: /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify -ts_type cn --args -fp_trap
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > warning: File "/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.so.6.0.24-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
> > To enable execution of this file add
> >         add-auto-load-safe-path /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.so.6.0.24-gdb.py
> > line to your configuration file "/raid/home/sajid/.gdbinit".
> > To completely disable this security protection add
> >         set auto-load safe-path /
> > line to your configuration file "/raid/home/sajid/.gdbinit".
> > For more information about this security protection see the
> > "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
> >         info "(gdb)Auto-loading safe path"
> > Solving a linear TS problem on 1 processor
> > mx : 512, my: 512 lambda : 1.239840e-10
> > 
> > Program received signal SIGFPE, Arithmetic exception.
> > __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
> >     at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> > 1978    /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c: No such file or directory.
> > Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7.x86_64 lapack-3.4.2-8.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libgfortran-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 systemd-libs-219-42.el7_4.4.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> > (gdb) bt
> > #0  __muldc3 (a=-6.6364040265716871e-306, b=1.1689456061105587e-305, c=-0.0024992568840190117, d=0.024886737403015963)
> >     at /raid/home/sajid/packages/spack/var/spack/stage/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/gcc-7.3.0/libgcc/libgcc2.c:1978
> > #1  0x00007ffff630fe87 in MatSolve_SeqAIJ_NaturalOrdering ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #2  0x00007ffff5ba61a8 in MatSolve ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #3  0x00007ffff6bc8a55 in PCApply_ILU ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #4  0x00007ffff6cde6eb in PCApply ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #5  0x00007ffff6e3ad4a in KSP_PCApply ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #6  0x00007ffff6e3bc36 in KSPInitialResidual ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #7  0x00007ffff6dc0736 in KSPSolve_GMRES ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #8  0x00007ffff6e1158e in KSPSolve ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #9  0x00007ffff6fac311 in SNESSolve_KSPONLY ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #10 0x00007ffff6f346c7 in SNESSolve ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #11 0x00007ffff714d58b in TSTheta_SNESSolve ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #12 0x00007ffff714df63 in TSStep_Theta ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #13 0x00007ffff7053ebe in TSStep ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > ---Type <return> to continue, or q <return> to quit---
> > #14 0x00007ffff70560bc in TSSolve ()
> >    from /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj/lib/libpetsc.so.3.010
> > #15 0x000000000040271d in main (argc=5, argv=0x7fffffffce38) at /raid/home/sajid/packages/xwp_petsc/2d/free_space/ex_modify.c:183
> > 
> 


From yann.jobic at univ-amu.fr  Sat Jan  5 03:03:59 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Sat, 5 Jan 2019 10:03:59 +0100
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
	<CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
Message-ID: <498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>


On 05/01/2019 02:36, Matthew Knepley wrote:
> On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Dear Petsc Users,
>
>     I'm using DMPlexCreateFromCellList to create my DM. I would like
>     to have
>     an order 2 geometry.
>
>     It's working fine in 2D for elements of type Q9.
>
>
> I do not see how that is possible. The topology does not work that 
> way, and there
> is no way a 9 vertex quad was interpolated, so something else is 
> happening.
>
> Moreover, anything you input still has affine geometry. Toby has been 
> working on
> non-affine geometry, and all the code is there, but right now you have 
> to build and
> populate the coordinate space by hand. What this means is that you 
> create a
> quadratic space for the coordinate DM (rather than the default 
> linear), get the
> global coordinate vector out, and stick in the Q9 coordinates. The 
> topology is still
> the same no matter what coordinates you read in.

I get the idea. So at the beginning i start with the Q4/H8 dmplex, then 
i fill from the global coordinate vector the missing coordinates, and 
attach this new coordinate vector to the DM. Then the interpolation will 
be correct ? Can i use DMproject and all the petsc functions ?

In order to output the results, i've got to write my own vtk interface 
no ? (by splitting the Q9/H27 in 4xQ4 or 8xH8).

Or is it working with the hdf5 one ?

Does the p4est interface work with this setup ?

>
> We will get around to making an interface for this soon.

Great !

Thanks,

Yann

>
> ? Thanks,
>
> ? ? ?Matt
>
>     I checked that it's working correctly by using
>     DMPlexComputeCellGeometryFEM, and compute the value of the
>     determinant
>     (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>
>     I can import H8 elements, it's working fine.
>
>     But for H27 element, it's not working. I really don't know how to
>     order
>     my vertex in the cells array. So far, the determinant is zero...
>
>     I don't know where to look in order to find this information. I tried
>     the function DMPlexGetRawFaces_Internal of the file
>
>     https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>
>     but it didn't help me.
>
>     Could you please point me where to look ?
>
>     Thanks, and happy new year !
>
>     Yann
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190105/c0efa278/attachment.html>

From knepley at gmail.com  Sat Jan  5 06:23:11 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 5 Jan 2019 07:23:11 -0500
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
	<CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
	<498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>
Message-ID: <CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>

On Sat, Jan 5, 2019 at 4:04 AM Yann Jobic <yann.jobic at univ-amu.fr> wrote:

>
> On 05/01/2019 02:36, Matthew Knepley wrote:
>
> On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear Petsc Users,
>>
>> I'm using DMPlexCreateFromCellList to create my DM. I would like to have
>> an order 2 geometry.
>>
>> It's working fine in 2D for elements of type Q9.
>>
>
> I do not see how that is possible. The topology does not work that way,
> and there
> is no way a 9 vertex quad was interpolated, so something else is happening.
>
> Moreover, anything you input still has affine geometry. Toby has been
> working on
> non-affine geometry, and all the code is there, but right now you have to
> build and
> populate the coordinate space by hand. What this means is that you create a
> quadratic space for the coordinate DM (rather than the default linear),
> get the
> global coordinate vector out, and stick in the Q9 coordinates. The
> topology is still
> the same no matter what coordinates you read in.
>
> I get the idea. So at the beginning i start with the Q4/H8 dmplex, then i
> fill from the global coordinate vector the missing coordinates, and attach
> this new coordinate vector to the DM.
>
Yes.

> Then the interpolation will be correct ?
>
This is the tricky part. What you likely have to do is:

  1) Build the topology
  2) Run over the cells
  3) For each cell, put in the coordinates using DMPlexVecSetClosure().
This is redundant, but the only way
      you can connect with the Q9 format.

> Can i use DMproject and all the petsc functions ?
>
This is the idea. The amount of testing is so far almost nothing.

> In order to output the results, i've got to write my own vtk interface no
> ? (by splitting the Q9/H27 in 4xQ4 or 8xH8).
>
That is a good question. Right now, it will just output Q4. However,
compared to everything else, writing Q9 should
be comparatively easy.

> Or is it working with the hdf5 one ?
>
I would probably write code to to Xdmf through HDF5 rather than straight
VTK, but we will end up having both.

> Does the p4est interface work with this setup ?
>
It will. I do not know if it currently does. We have to check.

  Thanks,

    Matt

>
> We will get around to making an interface for this soon.
>
> Great !
>
> Thanks,
>
> Yann
>
>
>   Thanks,
>
>      Matt
>
>
>> I checked that it's working correctly by using
>> DMPlexComputeCellGeometryFEM, and compute the value of the determinant
>> (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>>
>> I can import H8 elements, it's working fine.
>>
>> But for H27 element, it's not working. I really don't know how to order
>> my vertex in the cells array. So far, the determinant is zero...
>>
>> I don't know where to look in order to find this information. I tried
>> the function DMPlexGetRawFaces_Internal of the file
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>>
>> but it didn't help me.
>>
>> Could you please point me where to look ?
>>
>> Thanks, and happy new year !
>>
>> Yann
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190105/5261dcde/attachment.html>

From chang.liu at utexas.edu  Sun Jan  6 15:12:03 2019
From: chang.liu at utexas.edu (Chang Liu)
Date: Sun, 6 Jan 2019 15:12:03 -0600
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
Message-ID: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>

Hi All,
Recently, when I am upgrading our code from 3.8 to 3.10, it always runs
into error during VecGetArrayF90 call (it is Fortran90-based code). The
reason we use is to access the array for our user-defined matrix-vector
multiplication subroutine, for example:

*  subroutine mymult(A,x,y,loco_ierr) *
*   !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
*    use mat_vec_mult,only:matvec*
*    implicit none*
*    Mat,intent(in)::A ! required by PETSc*
*    Vec,intent(in)::x*
*    Vec,intent(out)::y*
*    PetscErrorCode,intent(out)::loco_ierr*
*    PetscScalar,pointer::xx(:),yy(:)*

*    call VecGetArrayReadF90(x,xx,loco_ierr)*
*    call VecGetArrayF90(y,yy,loco_ierr)*
*    call matvec(xx,yy)*
*    call VecRestoreArrayReadF90(x,xx,loco_ierr)*
*    call VecRestoreArrayF90(y,yy,loco_ierr)*
*    return*
* end subroutine mymult*

I checked the change log on the website and saw a statement:
TAO:

   - Added VecLock{Push|Pop} calls around user callbacks; use of
   VecGetArray in user callbacks is now prohibited.

Is there any relation to my issue? All the online PETSc examples and
description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I
tried to add/use VecLockPop/Push, but still doesn't work.

I am confused and not sure what is the problem?
Could I get some help?
Thanks,
-- 
Chang Liu
PhD candidate
Dept. of Electrical & Computer Engineering
Ph: 765-7143357
Email: chang.liu at utexas.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190106/b7011833/attachment.html>

From balay at mcs.anl.gov  Sun Jan  6 16:45:42 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sun, 6 Jan 2019 22:45:42 +0000
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901061641440.16192@sb>

This code is missing:

#include <petsc/finclude/petscvec.h>
      use petscvec

Check:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html

Satish

On Sun, 6 Jan 2019, Chang Liu via petsc-users wrote:

> Hi All,
> Recently, when I am upgrading our code from 3.8 to 3.10, it always runs
> into error during VecGetArrayF90 call (it is Fortran90-based code). The
> reason we use is to access the array for our user-defined matrix-vector
> multiplication subroutine, for example:
> 
> *  subroutine mymult(A,x,y,loco_ierr) *
> *   !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
> *    use mat_vec_mult,only:matvec*
> *    implicit none*
> *    Mat,intent(in)::A ! required by PETSc*
> *    Vec,intent(in)::x*
> *    Vec,intent(out)::y*
> *    PetscErrorCode,intent(out)::loco_ierr*
> *    PetscScalar,pointer::xx(:),yy(:)*
> 
> *    call VecGetArrayReadF90(x,xx,loco_ierr)*
> *    call VecGetArrayF90(y,yy,loco_ierr)*
> *    call matvec(xx,yy)*
> *    call VecRestoreArrayReadF90(x,xx,loco_ierr)*
> *    call VecRestoreArrayF90(y,yy,loco_ierr)*
> *    return*
> * end subroutine mymult*
> 
> I checked the change log on the website and saw a statement:
> TAO:
> 
>    - Added VecLock{Push|Pop} calls around user callbacks; use of
>    VecGetArray in user callbacks is now prohibited.
> 
> Is there any relation to my issue? All the online PETSc examples and
> description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I
> tried to add/use VecLockPop/Push, but still doesn't work.
> 
> I am confused and not sure what is the problem?
> Could I get some help?
> Thanks,
> 


From chang.liu at utexas.edu  Sun Jan  6 16:54:39 2019
From: chang.liu at utexas.edu (Chang Liu)
Date: Sun, 6 Jan 2019 16:54:39 -0600
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <alpine.LFD.2.21.1901061641440.16192@sb>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
	<alpine.LFD.2.21.1901061641440.16192@sb>
Message-ID: <CAKVVbQ8SRmibZbN5=4MvFmkkJx5pAvjHGSbVL7wf_eXGtv5HXA@mail.gmail.com>

Hi Satish,
I included that part in the module... At the beginning, I have:
#include "petsc/finclude/petscsys.h"
#include "petsc/finclude/petscvec.h"
#include "petsc/finclude/petscmat.h"
#include "petsc/finclude/petscpc.h"
#include "petsc/finclude/petscksp.h"
use petscksp

Sorry, I didnt mention at first...
Thanks,
Chang

On Sun, Jan 6, 2019 at 4:45 PM Balay, Satish <balay at mcs.anl.gov> wrote:

> This code is missing:
>
> #include <petsc/finclude/petscvec.h>
>       use petscvec
>
> Check:
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html
>
> Satish
>
> On Sun, 6 Jan 2019, Chang Liu via petsc-users wrote:
>
> > Hi All,
> > Recently, when I am upgrading our code from 3.8 to 3.10, it always runs
> > into error during VecGetArrayF90 call (it is Fortran90-based code). The
> > reason we use is to access the array for our user-defined matrix-vector
> > multiplication subroutine, for example:
> >
> > *  subroutine mymult(A,x,y,loco_ierr) *
> > *   !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
> > *    use mat_vec_mult,only:matvec*
> > *    implicit none*
> > *    Mat,intent(in)::A ! required by PETSc*
> > *    Vec,intent(in)::x*
> > *    Vec,intent(out)::y*
> > *    PetscErrorCode,intent(out)::loco_ierr*
> > *    PetscScalar,pointer::xx(:),yy(:)*
> >
> > *    call VecGetArrayReadF90(x,xx,loco_ierr)*
> > *    call VecGetArrayF90(y,yy,loco_ierr)*
> > *    call matvec(xx,yy)*
> > *    call VecRestoreArrayReadF90(x,xx,loco_ierr)*
> > *    call VecRestoreArrayF90(y,yy,loco_ierr)*
> > *    return*
> > * end subroutine mymult*
> >
> > I checked the change log on the website and saw a statement:
> > TAO:
> >
> >    - Added VecLock{Push|Pop} calls around user callbacks; use of
> >    VecGetArray in user callbacks is now prohibited.
> >
> > Is there any relation to my issue? All the online PETSc examples and
> > description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I
> > tried to add/use VecLockPop/Push, but still doesn't work.
> >
> > I am confused and not sure what is the problem?
> > Could I get some help?
> > Thanks,
> >
>
>

-- 
Chang Liu
PhD candidate
Dept. of Electrical & Computer Engineering
Ph: 765-7143357
Email: chang.liu at utexas.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190106/59316f62/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan  6 17:05:06 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 6 Jan 2019 23:05:06 +0000
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
Message-ID: <6EF96B5B-378B-48F4-AEB8-0136E7B4A061@anl.gov>


   What is the error message, cut and paste the entire message?

   Barry


> On Jan 6, 2019, at 3:12 PM, Chang Liu via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi All,
> Recently, when I am upgrading our code from 3.8 to 3.10, it always runs into error during VecGetArrayF90 call (it is Fortran90-based code). The reason we use is to access the array for our user-defined matrix-vector multiplication subroutine, for example:
> 
>   subroutine mymult(A,x,y,loco_ierr) 
>    !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>     use mat_vec_mult,only:matvec
>     implicit none
>     Mat,intent(in)::A ! required by PETSc
>     Vec,intent(in)::x
>     Vec,intent(out)::y
>     PetscErrorCode,intent(out)::loco_ierr
>     PetscScalar,pointer::xx(:),yy(:)
> 
>     call VecGetArrayReadF90(x,xx,loco_ierr)
>     call VecGetArrayF90(y,yy,loco_ierr)
>     call matvec(xx,yy)
>     call VecRestoreArrayReadF90(x,xx,loco_ierr)
>     call VecRestoreArrayF90(y,yy,loco_ierr)
>     return
>  end subroutine mymult
> 
> I checked the change log on the website and saw a statement:
> TAO:
> 
> 	? Added VecLock{Push|Pop} calls around user callbacks; use of VecGetArray in user callbacks is now prohibited.
> Is there any relation to my issue? All the online PETSc examples and description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I tried to add/use VecLockPop/Push, but still doesn't work.
> 
> I am confused and not sure what is the problem?
> Could I get some help? 
> Thanks,
> -- 
> Chang Liu
> PhD candidate
> Dept. of Electrical & Computer Engineering
> Ph: 765-7143357
> Email: chang.liu at utexas.edu


From eijkhout at tacc.utexas.edu  Sun Jan  6 17:21:26 2019
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Sun, 6 Jan 2019 23:21:26 +0000
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <6EF96B5B-378B-48F4-AEB8-0136E7B4A061@anl.gov>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
	<6EF96B5B-378B-48F4-AEB8-0136E7B4A061@anl.gov>
Message-ID: <816928C7-38C2-4EC5-A4D4-483CADBAC030@tacc.utexas.edu>

Hey Barry,

Chang is one of my users and I?m out of my depth here:

> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Corrupt argument:

> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1

> [0]PETSC ERROR: #1 VecGetArray() line 1578 in

It looks like VecGetArrayF90 is no longer allowed in callbacks, which is what seems to be happening here.

Victor.


On Jan 6, 2019, at 5:05 PM, Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


  What is the error message, cut and paste the entire message?

  Barry


On Jan 6, 2019, at 3:12 PM, Chang Liu via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

Hi All,
Recently, when I am upgrading our code from 3.8 to 3.10, it always runs into error during VecGetArrayF90 call (it is Fortran90-based code). The reason we use is to access the array for our user-defined matrix-vector multiplication subroutine, for example:

 subroutine mymult(A,x,y,loco_ierr)
  !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   use mat_vec_mult,only:matvec
   implicit none
   Mat,intent(in)::A ! required by PETSc
   Vec,intent(in)::x
   Vec,intent(out)::y
   PetscErrorCode,intent(out)::loco_ierr
   PetscScalar,pointer::xx(:),yy(:)

   call VecGetArrayReadF90(x,xx,loco_ierr)
   call VecGetArrayF90(y,yy,loco_ierr)
   call matvec(xx,yy)
   call VecRestoreArrayReadF90(x,xx,loco_ierr)
   call VecRestoreArrayF90(y,yy,loco_ierr)
   return
end subroutine mymult

I checked the change log on the website and saw a statement:
TAO:

? Added VecLock{Push|Pop} calls around user callbacks; use of VecGetArray in user callbacks is now prohibited.
Is there any relation to my issue? All the online PETSc examples and description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I tried to add/use VecLockPop/Push, but still doesn't work.

I am confused and not sure what is the problem?
Could I get some help?
Thanks,
--
Chang Liu
PhD candidate
Dept. of Electrical & Computer Engineering
Ph: 765-7143357
Email: chang.liu at utexas.edu<mailto:chang.liu at utexas.edu>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190106/0d4e81c3/attachment-0001.html>

From bsmith at mcs.anl.gov  Sun Jan  6 17:24:56 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 6 Jan 2019 23:24:56 +0000
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <816928C7-38C2-4EC5-A4D4-483CADBAC030@tacc.utexas.edu>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
	<6EF96B5B-378B-48F4-AEB8-0136E7B4A061@anl.gov>
	<816928C7-38C2-4EC5-A4D4-483CADBAC030@tacc.utexas.edu>
Message-ID: <1494C924-6A85-48D4-9F56-F31127ED7E17@mcs.anl.gov>



> On Jan 6, 2019, at 5:21 PM, Victor Eijkhout <eijkhout at tacc.utexas.edu> wrote:
> 
> Hey Barry,
> 
> Chang is one of my users and I?m out of my depth here:
> 
> > [0]PETSC ERROR: --------------------- Error Message
> > --------------------------------------------------------------
> > [0]PETSC ERROR: Corrupt argument:
> 
> > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1
> 
> > [0]PETSC ERROR: #1 VecGetArray() line 1578 in
> 
> It looks like VecGetArrayF90 is no longer allowed in callbacks, which is what seems to be happening here.
> 
> Victor.
> 
> 
>> On Jan 6, 2019, at 5:05 PM, Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> 
>>   What is the error message, cut and paste the entire message?
>> 
>>   Barry
>> 
>> 
>>> On Jan 6, 2019, at 3:12 PM, Chang Liu via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi All,
>>> Recently, when I am upgrading our code from 3.8 to 3.10, it always runs into error during VecGetArrayF90 call (it is Fortran90-based code). The reason we use is to access the array for our user-defined matrix-vector multiplication subroutine, for example:
>>> 
>>>  subroutine mymult(A,x,y,loco_ierr) 
>>>   !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>    use mat_vec_mult,only:matvec
>>>    implicit none
>>>    Mat,intent(in)::A ! required by PETSc
>>>    Vec,intent(in)::x
>>>    Vec,intent(out)::y

   This is wrong. y is not returned from your subroutine, it is passed into the subroutine and hence is intent(in). 

   Hopefully this resolves your problem, otherwise let us know.

   Barry

>>>    PetscErrorCode,intent(out)::loco_ierr
>>>    PetscScalar,pointer::xx(:),yy(:)
>>> 
>>>    call VecGetArrayReadF90(x,xx,loco_ierr)
>>>    call VecGetArrayF90(y,yy,loco_ierr)
>>>    call matvec(xx,yy)
>>>    call VecRestoreArrayReadF90(x,xx,loco_ierr)
>>>    call VecRestoreArrayF90(y,yy,loco_ierr)
>>>    return
>>> end subroutine mymult
>>> 
>>> I checked the change log on the website and saw a statement:
>>> TAO:
>>> 
>>> ? Added VecLock{Push|Pop} calls around user callbacks; use of VecGetArray in user callbacks is now prohibited.
>>> Is there any relation to my issue? All the online PETSc examples and description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I tried to add/use VecLockPop/Push, but still doesn't work.
>>> 
>>> I am confused and not sure what is the problem?
>>> Could I get some help? 
>>> Thanks,
>>> -- 
>>> Chang Liu
>>> PhD candidate
>>> Dept. of Electrical & Computer Engineering
>>> Ph: 765-7143357
>>> Email: chang.liu at utexas.edu
>> 
> 


From chang.liu at utexas.edu  Sun Jan  6 18:11:26 2019
From: chang.liu at utexas.edu (Chang Liu)
Date: Sun, 6 Jan 2019 18:11:26 -0600
Subject: [petsc-users] VecGetArrayF90 in PETSc/3.10 (Fortran)
In-Reply-To: <1494C924-6A85-48D4-9F56-F31127ED7E17@mcs.anl.gov>
References: <CAKVVbQ-H5TsMx-eW80Awf7Y7yS748icpvmw_L4FKD0CocQfiCg@mail.gmail.com>
	<6EF96B5B-378B-48F4-AEB8-0136E7B4A061@anl.gov>
	<816928C7-38C2-4EC5-A4D4-483CADBAC030@tacc.utexas.edu>
	<1494C924-6A85-48D4-9F56-F31127ED7E17@mcs.anl.gov>
Message-ID: <CAKVVbQ9h2eNERDHYSc1qMGZd+0-DWWAJUwObUsGUNP7PqC=oyA@mail.gmail.com>

Hi Smith,

I tried and it works. Thank you so much! It is weird that this bug hides so
well since petsc/3.4,3.5 (5-6 years ago) in our code; now from petsc/3.9,
it shows up.....

Thank you guys for all your help~
Best,
Chang

On Sun, Jan 6, 2019 at 5:24 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 6, 2019, at 5:21 PM, Victor Eijkhout <eijkhout at tacc.utexas.edu>
> wrote:
> >
> > Hey Barry,
> >
> > Chang is one of my users and I?m out of my depth here:
> >
> > > [0]PETSC ERROR: --------------------- Error Message
> > > --------------------------------------------------------------
> > > [0]PETSC ERROR: Corrupt argument:
> >
> > > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1
> >
> > > [0]PETSC ERROR: #1 VecGetArray() line 1578 in
> >
> > It looks like VecGetArrayF90 is no longer allowed in callbacks, which is
> what seems to be happening here.
> >
> > Victor.
> >
> >
> >> On Jan 6, 2019, at 5:05 PM, Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>
> >>
> >>   What is the error message, cut and paste the entire message?
> >>
> >>   Barry
> >>
> >>
> >>> On Jan 6, 2019, at 3:12 PM, Chang Liu via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>>
> >>> Hi All,
> >>> Recently, when I am upgrading our code from 3.8 to 3.10, it always
> runs into error during VecGetArrayF90 call (it is Fortran90-based code).
> The reason we use is to access the array for our user-defined matrix-vector
> multiplication subroutine, for example:
> >>>
> >>>  subroutine mymult(A,x,y,loco_ierr)
> >>>   !!!!!!!!!!!!!!! matrix-vector multiplication  y=A.x
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> >>>    use mat_vec_mult,only:matvec
> >>>    implicit none
> >>>    Mat,intent(in)::A ! required by PETSc
> >>>    Vec,intent(in)::x
> >>>    Vec,intent(out)::y
>
>    This is wrong. y is not returned from your subroutine, it is passed
> into the subroutine and hence is intent(in).
>
>    Hopefully this resolves your problem, otherwise let us know.
>
>    Barry
>
> >>>    PetscErrorCode,intent(out)::loco_ierr
> >>>    PetscScalar,pointer::xx(:),yy(:)
> >>>
> >>>    call VecGetArrayReadF90(x,xx,loco_ierr)
> >>>    call VecGetArrayF90(y,yy,loco_ierr)
> >>>    call matvec(xx,yy)
> >>>    call VecRestoreArrayReadF90(x,xx,loco_ierr)
> >>>    call VecRestoreArrayF90(y,yy,loco_ierr)
> >>>    return
> >>> end subroutine mymult
> >>>
> >>> I checked the change log on the website and saw a statement:
> >>> TAO:
> >>>
> >>> ? Added VecLock{Push|Pop} calls around user callbacks; use of
> VecGetArray in user callbacks is now prohibited.
> >>> Is there any relation to my issue? All the online PETSc examples and
> description for VecGetArray/VecGetArrayF90 are the same as PETSc/3.8.  I
> tried to add/use VecLockPop/Push, but still doesn't work.
> >>>
> >>> I am confused and not sure what is the problem?
> >>> Could I get some help?
> >>> Thanks,
> >>> --
> >>> Chang Liu
> >>> PhD candidate
> >>> Dept. of Electrical & Computer Engineering
> >>> Ph: 765-7143357
> >>> Email: chang.liu at utexas.edu
> >>
> >
>
>

-- 
Chang Liu
PhD candidate
Dept. of Electrical & Computer Engineering
Ph: 765-7143357
Email: chang.liu at utexas.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190106/e6cfca6b/attachment.html>

From koshkarov.alexandr at usask.ca  Sun Jan  6 19:27:41 2019
From: koshkarov.alexandr at usask.ca (Koshkarov, Oleksandr)
Date: Mon, 7 Jan 2019 01:27:41 +0000
Subject: [petsc-users] Solve ODE in backward in time
Message-ID: <YTOPR0101MB0713EF23D119977DA3B3C79BF0890@YTOPR0101MB0713.CANPRD01.PROD.OUTLOOK.COM>

Hello all,

Can TSSolve be used to solve ODE backward in time? for example from t=10 upto t=5 ? Google search reveals to use TSAdjointSolve, but by reading manual, It looks it is used for sensitivity analysis which I do not need. And I am not sure how to use it to just integrate ODE back in time. I tried to setup final time and/or time step to be negative without success.

Thank you and Happy New Year,

Oleksandr.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/11fd48fd/attachment.html>

From jed at jedbrown.org  Sun Jan  6 21:02:38 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 06 Jan 2019 20:02:38 -0700
Subject: [petsc-users] Solve ODE in backward in time
In-Reply-To: <YTOPR0101MB0713EF23D119977DA3B3C79BF0890@YTOPR0101MB0713.CANPRD01.PROD.OUTLOOK.COM>
References: <YTOPR0101MB0713EF23D119977DA3B3C79BF0890@YTOPR0101MB0713.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <87h8elp3u9.fsf@jedbrown.org>

Is it a problem to do the change of variables in your problem definition?

"Koshkarov, Oleksandr via petsc-users" <petsc-users at mcs.anl.gov> writes:

> Hello all,
>
> Can TSSolve be used to solve ODE backward in time? for example from t=10 upto t=5 ? Google search reveals to use TSAdjointSolve, but by reading manual, It looks it is used for sensitivity analysis which I do not need. And I am not sure how to use it to just integrate ODE back in time. I tried to setup final time and/or time step to be negative without success.
>
> Thank you and Happy New Year,
>
> Oleksandr.

From yann.jobic at univ-amu.fr  Mon Jan  7 04:09:25 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Mon, 7 Jan 2019 11:09:25 +0100
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
	<CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
	<498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>
	<CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>
Message-ID: <fc11a823-39df-9fbb-7b54-d317fba2cf71@univ-amu.fr>


Le 05/01/2019 ? 13:23, Matthew Knepley a ?crit?:
> On Sat, Jan 5, 2019 at 4:04 AM Yann Jobic <yann.jobic at univ-amu.fr 
> <mailto:yann.jobic at univ-amu.fr>> wrote:
>
>
>     On 05/01/2019 02:36, Matthew Knepley wrote:
>>     On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users
>>     <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>
>>         Dear Petsc Users,
>>
>>         I'm using DMPlexCreateFromCellList to create my DM. I would
>>         like to have
>>         an order 2 geometry.
>>
>>         It's working fine in 2D for elements of type Q9.
>>
>>
>>     I do not see how that is possible. The topology does not work
>>     that way, and there
>>     is no way a 9 vertex quad was interpolated, so something else is
>>     happening.
>>
>>     Moreover, anything you input still has affine geometry. Toby has
>>     been working on
>>     non-affine geometry, and all the code is there, but right now you
>>     have to build and
>>     populate the coordinate space by hand. What this means is that
>>     you create a
>>     quadratic space for the coordinate DM (rather than the default
>>     linear), get the
>>     global coordinate vector out, and stick in the Q9 coordinates.
>>     The topology is still
>>     the same no matter what coordinates you read in.
>
>     I get the idea. So at the beginning i start with the Q4/H8 dmplex,
>     then i fill from the global coordinate vector the missing
>     coordinates, and attach this new coordinate vector to the DM.
>
> Yes.
In order to set up a quadratic space for my DM, i should use 
PetscDualSpaceSetOrder no ?

And then attach this space to my DM ?

When i add the missing vertex coordinates, the order should not matter 
no ? (it should at the next step with the DMPlexVecSetClosure)

I'll try to build a laplacian test case, with analytical solution in FEM 
in order to test it.

Thanks,

Yann

>     Then the interpolation will be correct ?
>
> This is the tricky part. What you likely have to do is:
>
> ? 1) Build the topology
> ? 2) Run over the cells
> ? 3) For each cell, put in the coordinates using 
> DMPlexVecSetClosure(). This is redundant, but the only way
> ? ? ? you can connect with the Q9 format.
>
>     Can i use DMproject and all the petsc functions ?
>
> This is the idea. The amount of testing is so far almost nothing.
>
>     In order to output the results, i've got to write my own vtk
>     interface no ? (by splitting the Q9/H27 in 4xQ4 or 8xH8).
>
> That is a good question. Right now, it will just output Q4. However, 
> compared to everything else, writing Q9 should
> be comparatively easy.
>
>     Or is it working with the hdf5 one ?
>
> I would probably write code to to Xdmf through HDF5 rather than 
> straight VTK, but we will end up having both.
>
>     Does the p4est interface work with this setup ?
>
> It will. I do not know if it currently does. We have to check.
>
> ? Thanks,
>
> ? ? Matt
>
>>
>>     We will get around to making an interface for this soon.
>
>     Great !
>
>     Thanks,
>
>     Yann
>
>>
>>     ? Thanks,
>>
>>     ? ? ?Matt
>>
>>         I checked that it's working correctly by using
>>         DMPlexComputeCellGeometryFEM, and compute the value of the
>>         determinant
>>         (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>>
>>         I can import H8 elements, it's working fine.
>>
>>         But for H27 element, it's not working. I really don't know
>>         how to order
>>         my vertex in the cells array. So far, the determinant is zero...
>>
>>         I don't know where to look in order to find this information.
>>         I tried
>>         the function DMPlexGetRawFaces_Internal of the file
>>
>>         https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>>
>>         but it didn't help me.
>>
>>         Could you please point me where to look ?
>>
>>         Thanks, and happy new year !
>>
>>         Yann
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>

-- 
___________________________

Yann JOBIC
HPC engineer
IUSTI-CNRS UMR 7343 - Polytech Marseille
Technop?le de Ch?teau Gombert
5 rue Enrico Fermi
13453 Marseille cedex 13
Tel : (33) 4 91 10 69 43
Fax : (33) 4 91 10 69 69

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/47c5257a/attachment-0001.html>

From dave.mayhem23 at gmail.com  Mon Jan  7 07:28:10 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Mon, 7 Jan 2019 14:28:10 +0100
Subject: [petsc-users] EGU 2019 session: Advances in Numerical Modelling of
 Geological Processes
In-Reply-To: <MW2PR02MB3754C064F766F799D4E95302AD8F0@MW2PR02MB3754.namprd02.prod.outlook.com>
References: <MW2PR02MB3754317871563F6F5D0D8E2FADB90@MW2PR02MB3754.namprd02.prod.outlook.com>
	<MW2PR02MB3754C064F766F799D4E95302AD8F0@MW2PR02MB3754.namprd02.prod.outlook.com>
Message-ID: <CAJ98EDohapN9E+JNUh2NMSg0iKbb+ZYOpYw-uH7dd9FY7muc8A@mail.gmail.com>

This EGU session (see description and link below) may be of interest to
some PETSc users.

Cheers,
  Dave


---------- Forwarded message ---------
From: Ludovic Raess <lraess at stanford.edu>
Date: Sat, 5 Jan 2019 at 15:16
Subject: [Myres_all] REMINDER - EGU 2019 session: Advances in Numerical
Modelling of Geological Processes
To: Ludovic Raess <lraess at stanford.edu>


Dear colleagues,


EGU 2019 will take place in Vienna, 7-12 April 2019.  The abstract
submission deadline is approaching (10 January 2019).


For those interested in the application and the development of numerical
models for the study of geological processes, feel free to join:


Advances in Numerical Modelling of Geological Processes
<https://meetingorganizer.copernicus.org/EGU2019/session/30976>


Session details can be find below.


Cheers,


T. Duretz, L. R?ss, B. J. P. Kaus and D. A. May


-----------------------------------------------------------------------------------------------------------


*Advances in Numerical Modelling of Geological Processes * [
GD8.1/EMRP1.81/SM7.6/TS11.6]



Geological and geophysical data provide quantitative information which
permit the advancement of our understanding of the present, and past,
interior of the Earth. Examples of such processes span from the internal
structure of the Earth, plate kinematics, composition of geomaterials,
estimation of physical conditions and dating of key geological events,
thermal state of the Earth to more shallow processes such as reservoir
geomechanics, or nuclear waste storage.


A quantitative understanding of the dynamics and the feedbacks between
geological processes requires the integration of geological data with
process oriented numerical models. Innovative inverse methods, linking
forward dynamic models with observables, are topics of growing interest
within the community. Improving our knowledge of the governing physical
parameters can thus be addressed while reconciling models and observables.


Resolving the interactions between various processes occurring at scales
differing from each other over several orders of magnitude in space and
time represents a computational challenge. Hence, simulating such coupled,
nonlinear physics-based forward models requires both the development of new
approaches and the enhancement of established numerical schemes.


The majority of geological processes combine several physical mechanisms
such as hydrological, thermal, chemical and mechanical processes (e.g.
thermo-mechanical convection). Understanding the tight couplings among
those processes represents a challenging and essential research direction.
The development of novel numerical modelling approaches, which resolve
multi-physics feedbacks, is vital in order to provide accurate predictions
and gain deeper understanding of geological processes.


We invite contributions from the following two complementary themes:


#1 Computational advances associated with

- alternative spatial and/or temporal discretisations for existing
forward/inverse models

- scalable HPC implementations of new and existing methodologies (GPUs /
multi-core)

- solver and preconditioner developments

- code and methodology comparisons (?benchmarks?)

- open source implementations for the community


#2 Physics advances associated with

- development of partial differential equations to describe geological
processes

- inverse and adjoint-based methods

- numerical model validation through comparison with natural observations
and geophysical data

- scientific insights enabled by 2D and 3D modelling

- utilisation of coupled models to address nonlinear interactions

_______________________________________________
Myres_all mailing list
Myres_all at whoi.edu
http://mailman.whoi.edu/mailman/listinfo/myres_all
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/f843b205/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Mon Jan  7 09:05:57 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 7 Jan 2019 09:05:57 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <alpine.LFD.2.21.1901041940270.20930@sb>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
	<alpine.LFD.2.21.1901041940270.20930@sb>
Message-ID: <CAOGsD9hAd9qjwoB8_DekD9peTJHOrNNnOPa_TaDXAo_5CcQhFg@mail.gmail.com>

 >   Anyway the FPE occurs in MatSolve_SeqAIJ_NaturalOrdering which usually
indicates a zero on the diagonal of the matrix. Is that possible?
It looks like this is indeed the case here. Thanks for the hint.

@Satish Balay <balay at mcs.anl.gov> : I tried building with the patch and
don't see any difference. Do you want me to send you the config and build
logs to investigate further? Apart from the -g flag, as I've stated above
another bug is that petsc uses the system gdb (rhel-7) and not the gdb
associated with the gcc that was used to build petsc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/dce156e2/attachment.html>

From balay at mcs.anl.gov  Mon Jan  7 09:55:52 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 7 Jan 2019 15:55:52 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9hAd9qjwoB8_DekD9peTJHOrNNnOPa_TaDXAo_5CcQhFg@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
	<alpine.LFD.2.21.1901041940270.20930@sb>
	<CAOGsD9hAd9qjwoB8_DekD9peTJHOrNNnOPa_TaDXAo_5CcQhFg@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901070945020.16192@sb>

On Mon, 7 Jan 2019, Sajid Ali wrote:

> @Satish Balay <balay at mcs.anl.gov> : I tried building with the patch and
> don't see any difference. Do you want me to send you the config and build
> logs to investigate further?

Hm - you should see '-g' for 'petsc+debug'. Send the logs. 

I have - in /home/balay/spack/opt/spack/linux-fedora29-x86_64/gcc-8.2.1/petsc-3.10.3-q5v3hfhsecvhf3geffeyrm5j3bm6orup/.spack/build.out
>>>>>>>>
Compilers:
  C Compiler:         /home/petsc/soft/mpich-3.3b1/bin/mpicc    -g3 
  C++ Compiler:       /home/petsc/soft/mpich-3.3b1/bin/mpic++  -g      
  Fortran Compiler:   /home/petsc/soft/mpich-3.3b1/bin/mpif90   -g  
Linkers:
  Shared linker:   /home/petsc/soft/mpich-3.3b1/bin/mpicc  -shared    -g3
  Dynamic linker:   /home/petsc/soft/mpich-3.3b1/bin/mpicc  -shared    -g3
<<<<<

Note: the patch I sent earlier fails for 'petsc~debug' - but the
attached updated patch should work for it.

> Apart from the -g flag, as I've stated above
> another bug is that petsc uses the system gdb (rhel-7) and not the gdb
> associated with the gcc that was used to build petsc.

It uses gdb from your PATH - so you can setup your PATH to use the appropriate gdb.

Satish
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: spack-petsc-debug.patch
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/8d8762fb/attachment.ksh>

From tempohoper at gmail.com  Mon Jan  7 10:23:16 2019
From: tempohoper at gmail.com (Sal Am)
Date: Mon, 7 Jan 2019 16:23:16 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
	<CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
	<DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>
Message-ID: <CAE3k7C4L2_vvWJEnxmByfzVE8kD4D08V0zTh6dSZuRiWxf0u=A@mail.gmail.com>

>
> Recent versions of MUMPS have bugs when using SCALAPACK. You may also try
> to disable parallelism on the root node by using

tried that, does not seem to help... but it is odd. this is not a big
system, it is small and made for testing.

perhaps there is something wrong with the MPI....If anyone is using the
PBSPro for queueing system, could it be the incompatibility between the
MPICH and the PBS? Should one use OpenMPI?

On Thu, Jan 3, 2019 at 3:55 PM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> Recent versions of MUMPS have bugs when using SCALAPACK. You may also try
> to disable parallelism on the root node by using
>
>  -mat_mumps_icntl_13 1
>
> On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> You may try different matrix orderings, or try superlu_dist.
> Hong
>
>
> On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Error in external library
>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-22193
>>>
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [4]PETSC ERROR: Error in external library
>>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-24197
>>>
>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [16]PETSC ERROR: Error in external library
>>> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [20]PETSC ERROR: Error in external library
>>> [20]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [24]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [24]PETSC ERROR: Error in external library
>>> [24]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [24]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>> [24]PETSC ERROR: #4 PCSetUp() line 932 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [24]PETSC ERROR: #5 KSPSetUp() line 391 in
>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>> [24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [32]PETSC ERROR: Error in external library
>>> [32]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=0
>>>
>>> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [32]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>> [32]PETSC ERROR: #4 PCSetUp() line 932 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [32]PETSC ERROR: #5 KSPSetUp() line 391 in
>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>> [32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [8]PETSC ERROR: Error in external library
>>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-35665
>>>
>>> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>> [8]PETSC ERROR: #4 PCSetUp() line 932 in
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [12]PETSC ERROR: Error in external library
>>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFOG(1)=-13, INFO(2)=-16661
>>>
>>> [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>> Jan  3 04:32:43 2019
>>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=no --download-scalapack --download-superlu
>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>> [12]PETSC ERROR: #6 main() line 68 in
>>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>>> [24]PETSC ERROR: PETSc Option Table entries:
>>> [24]PETSC ERROR: -ksp_error_if_not_converged
>>> [24]PETSC ERROR: -ksp_max_it 1
>>> [24]PETSC ERROR: -ksp_monitor_true_residual
>>> [24]PETSC ERROR: -ksp_type richardson
>>> [24]PETSC ERROR: -log_view
>>> [24]PETSC ERROR: -mat_mumps_icntl_14 100
>>> [24]PETSC ERROR: -pc_factor_mat_solver_type mumps
>>> [24]PETSC ERROR: -pc_type lu
>>> [24]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>>
>>> Sorry to bother again, but increasing ICNTL(14) does not seem to have
>>> helped... I am not sure what I am doing wrong exactly. As you can see I
>>> have increased it by 100%, prior, I tried to increase it with 40-70% but
>>> none worked.
>>>
>>
>> I think this is for the MUMPS list, but it appears you have a lot of
>> fill-in for this matrix and it might be too much for a direct solver.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi I am getting an error using MUMPS.
>>>>> How I run it:
>>>>> bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu
>>>>> -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual
>>>>> -log_view -ksp_error_if_not_converged
>>>>>
>>>>> The error output:
>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Error in external library
>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [1]PETSC ERROR: Error in external library
>>>>> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [1]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [6]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [6]PETSC ERROR: Error in external library
>>>>> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [8]PETSC ERROR: Error in external library
>>>>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [8]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: ---------------------
>>>>> Error Message --------------------------------------------------------------
>>>>> [15]PETSC ERROR: Error in external library
>>>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-36536
>>>>>
>>>>> [15]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-81813
>>>>>
>>>>> [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [2]PETSC ERROR: Error in external library
>>>>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [2]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [3]PETSC ERROR: Error in external library
>>>>> [3]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-28194
>>>>>
>>>>> [3]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [4]PETSC ERROR: Error in external library
>>>>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [4]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [5]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [5]PETSC ERROR: Error in external library
>>>>> [5]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-27685
>>>>>
>>>>> [5]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [5]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [7]PETSC ERROR: Error in external library
>>>>> [7]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [7]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13
>>>>> 2018
>>>>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>> [10]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [10]PETSC ERROR: Error in external library
>>>>> [10]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [10]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [11]PETSC ERROR: Error in external library
>>>>> [11]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-30519
>>>>>
>>>>> [11]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [12]PETSC ERROR: Error in external library
>>>>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-28435
>>>>>
>>>>> [12]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [13]PETSC ERROR: Error in external library
>>>>> [13]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-30110
>>>>>
>>>>> [13]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [14]PETSC ERROR: Error in external library
>>>>> [14]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [14]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed
>>>>> Dec 12 10:44:13 2018
>>>>> [14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [15]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>> [15]PETSC ERROR: #4 PCSetUp() line 932 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>> [15]PETSC ERROR: #5 KSPSetUp() line 391 in
>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [15]PETSC ERROR: #6 main() line 68 in
>>>>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>>>>> [15]PETSC ERROR: PETSc Option Table entries:
>>>>> [15]PETSC ERROR: -ksp_error_if_not_converged
>>>>> [15]PETSC ERROR: -ksp_max_it 1
>>>>> [15]PETSC ERROR: -ksp_monitor_true_residual
>>>>> [15]PETSC ERROR: -ksp_type richardson
>>>>> [15]PETSC ERROR: -log_view
>>>>> [15]PETSC ERROR: -pc_factor_mat_solver_type mumps
>>>>> [15]PETSC ERROR: -pc_type lu
>>>>> [15]PETSC ERROR: ----------------End of Error Message -------send
>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>
>>>>> From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
>>>>> Problem of workspace allocation of size INFO(2) during the
>>>>> factorization or solve steps. The size that the package tried to allocate
>>>>> with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is
>>>>> negative, then the size that the package requested is obtained by
>>>>> multiplying the absolute value of INFO(2) by 1 million. In general, the
>>>>> unit for INFO(2) is the number of scalar entries of  the type of the input
>>>>> matrix (real, complex, single or double precision).
>>>>>
>>>>> Not quite sure what to make of it as the first error indicates size
>>>>> INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
>>>>>
>>>> But you can see there are processes with negative INFO(2). According to
>>>> MUMPS manual, you can try to increase ICNTL(14).
>>>>
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/2bbc2041/attachment-0001.html>

From tempohoper at gmail.com  Mon Jan  7 10:31:25 2019
From: tempohoper at gmail.com (Sal Am)
Date: Mon, 7 Jan 2019 16:31:25 +0000
Subject: [petsc-users] Configuring PETSc with OpenMPI
Message-ID: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>

Added the log file.

>From OpenMPI:

> The only special configuration that you need to build PETSc is to ensure
> that Open MPI's wrapper compilers (i.e., mpicc and mpif77) are in your
> $PATH before running the PETSc configure.py script.
>
> PETSc should then automatically find Open MPI's wrapper compilers and
> correctly build itself using Open MPI.
>
The OpenMPI dir is on my PATH which contain mpicc and mpif77.

This is on a HPC, if that matters.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/4eda2ddc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2074271 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/4eda2ddc/attachment-0001.obj>

From balay at mcs.anl.gov  Mon Jan  7 10:39:41 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 7 Jan 2019 16:39:41 +0000
Subject: [petsc-users] Configuring PETSc with OpenMPI
In-Reply-To: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>
References: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901071038440.16192@sb>

Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-cumulus-hpc --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-mpi-dir=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/ --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake

' --with-cc=gcc --with-fc=gfortran --with-cxx=g++' prevents usage of mpicc etc - so remove these options when using with-mpi-dir

Or use:

--with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..

Satish


On Mon, 7 Jan 2019, Sal Am via petsc-users wrote:

> Added the log file.
> 
> >From OpenMPI:
> 
> > The only special configuration that you need to build PETSc is to ensure
> > that Open MPI's wrapper compilers (i.e., mpicc and mpif77) are in your
> > $PATH before running the PETSc configure.py script.
> >
> > PETSc should then automatically find Open MPI's wrapper compilers and
> > correctly build itself using Open MPI.
> >
> The OpenMPI dir is on my PATH which contain mpicc and mpif77.
> 
> This is on a HPC, if that matters.
> 


From sajidsyed2021 at u.northwestern.edu  Mon Jan  7 10:38:45 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Mon, 7 Jan 2019 10:38:45 -0600
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <alpine.LFD.2.21.1901070945020.16192@sb>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
	<alpine.LFD.2.21.1901041940270.20930@sb>
	<CAOGsD9hAd9qjwoB8_DekD9peTJHOrNNnOPa_TaDXAo_5CcQhFg@mail.gmail.com>
	<alpine.LFD.2.21.1901070945020.16192@sb>
Message-ID: <CAOGsD9h+_e-rvGPvf8wk-NUt5dCbk6WgzBt9c3xQjEsEdhZtHA@mail.gmail.com>

Hi Satish,

Please find attached the logs for the previous patch for petsc+debug.

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/e7b02def/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: spack-build.out
Type: application/octet-stream
Size: 184917 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/e7b02def/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 5509953 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/e7b02def/attachment-0001.bin>

From fuentesdt at gmail.com  Mon Jan  7 10:41:42 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Mon, 7 Jan 2019 10:41:42 -0600
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
Message-ID: <CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>

thanks Matt,

I posted a slightly modified example

https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c

and changes from the orginal

https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff


This is what I'm seeing

https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png


I'm refining based on this element centroid:

https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl

unrefined -
https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu

refined -
https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu


Maybe I don't understand the refinement algorithm, but the refinement is a
little offset from what I was expecting.




Also, can you pass an application context to the user refinement function ?

https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119



On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> Starting with TS example 45
>>
>> $ pwd
>> /opt/apps/PETSc/petsc-3.10.2
>> $ ls src/ts/examples/tutorials/ex45.c
>> src/ts/examples/tutorials/ex45.c
>>
>> petsc configured with: ./config/configure.py --with-shared-libraries
>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>> --download-ctetgen  --download-triangle --with-debugging=yes
>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>> --download-pnetcdf
>>
>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>> DMPlexSetRefinementFunction.
>> It generally seems to be working, except that the refined element is
>> slightly offset from what I was expecting.
>> Based on my application specific criteria, element id number 7 is flagged
>> to be refined by the DMPlexSetRefinementFunction but when I open in
>> paraview, it looks like element id number 8 is being refined. See attached
>> pic.
>>
>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>
>> Is it possible that the maxVolumes array is 'off by one' when transfering
>> to tetgen data structures somehow ?
>>
>
> I looked through and cannot see it by eye. Could you send me your modified
> example and I will walk through it with the
> debugger?
>
>   Thanks,
>
>      Matt
>
>
>>
>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>
>>
>>
>> (gdb) bt
>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>> dmRefined=0x7fffffffb938) at
>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>> dmRefined=0x7fffffffb938) at
>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>> dmf=0x7fffffffb938) at
>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/f79fe153/attachment.html>

From balay at mcs.anl.gov  Mon Jan  7 10:47:26 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 7 Jan 2019 16:47:26 +0000
Subject: [petsc-users] Question about correctly catching fp_trap
In-Reply-To: <CAOGsD9h+_e-rvGPvf8wk-NUt5dCbk6WgzBt9c3xQjEsEdhZtHA@mail.gmail.com>
References: <CAOGsD9g+pVTG70QuPVKDHiq2rq2PC3+NzfmKeWYiJzBv3rBahQ@mail.gmail.com>
	<05BAE1F2-B80D-4164-85EF-6E5C05D7FB11@anl.gov>
	<alpine.LFD.2.21.1901032038350.20930@sb>
	<2BDB5BC8-6BBD-4D4E-9FE7-0FF2AFA271D2@mcs.anl.gov>
	<CAOGsD9hvE7-0-BXG-fSKmPnRieYL1+jUX3oZHK_Aki6e2Ds+Jg@mail.gmail.com>
	<9F05D2A6-D406-4732-80E7-BE8268B7BC2A@mcs.anl.gov>
	<CAOGsD9jpHU3Brj8r6f1gbz34n7=DcRNVtLk0tN2zPDYFinMWyQ@mail.gmail.com>
	<CAOGsD9j71zpUve9F5_mNGk-wnW3RUdHTm=B9c0y3nkOPSA8Qkw@mail.gmail.com>
	<302B9D83-B36B-47C4-873E-1F993E2DF243@mcs.anl.gov>
	<alpine.LFD.2.21.1901041940270.20930@sb>
	<CAOGsD9hAd9qjwoB8_DekD9peTJHOrNNnOPa_TaDXAo_5CcQhFg@mail.gmail.com>
	<alpine.LFD.2.21.1901070945020.16192@sb>
	<CAOGsD9h+_e-rvGPvf8wk-NUt5dCbk6WgzBt9c3xQjEsEdhZtHA@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901071046130.16192@sb>

On Mon, 7 Jan 2019, Sajid Ali via petsc-users wrote:

> Hi Satish,
> 
> Please find attached the logs for the previous patch for petsc+debug.

>>>>>>>
Compilers:
  C Compiler:         /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc  -fPIC  -g3
  C++ Compiler:       /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpic++  -g   -fPIC
  Fortran Compiler:   /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpif90  -fPIC -g
Linkers:
  Shared linker:   /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc  -shared  -fPIC  -g3
  Dynamic linker:   /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc  -shared  -fPIC  -g3
<<<<<<<<


So the build did use '-g' flag - so it looks good to me..

Satish

From hzhang at mcs.anl.gov  Mon Jan  7 10:53:12 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Mon, 7 Jan 2019 16:53:12 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAE3k7C4L2_vvWJEnxmByfzVE8kD4D08V0zTh6dSZuRiWxf0u=A@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
	<CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
	<DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>
	<CAE3k7C4L2_vvWJEnxmByfzVE8kD4D08V0zTh6dSZuRiWxf0u=A@mail.gmail.com>
Message-ID: <CAGCphBtyp2jD96FZGz4hAZ_TyO7VBq98bQZpZUCOkMC-uU12sg@mail.gmail.com>

Sal:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using
tried that, does not seem to help... but it is odd. this is not a big system, it is small and made for testing.

How large is the matrix, its dimension and number of non-zeros?
Can you run it sequentially or with less number of processors?

perhaps there is something wrong with the MPI....If anyone is using the PBSPro for queueing system, could it be the incompatibility between the MPICH and the PBS? Should one use OpenMPI?
 Can you run your case on different machine?
Hong

On Thu, Jan 3, 2019 at 3:55 PM Stefano Zampini <stefano.zampini at gmail.com<mailto:stefano.zampini at gmail.com>> wrote:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using

 -mat_mumps_icntl_13 1

On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

You may try different matrix orderings, or try superlu_dist.
Hong


On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-22193

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-24197

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[24]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-35665

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[8]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-16661

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -ksp_error_if_not_converged
[24]PETSC ERROR: -ksp_max_it 1
[24]PETSC ERROR: -ksp_monitor_true_residual
[24]PETSC ERROR: -ksp_type richardson
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_mumps_icntl_14 100
[24]PETSC ERROR: -pc_factor_mat_solver_type mumps
[24]PETSC ERROR: -pc_type lu
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

Sorry to bother again, but increasing ICNTL(14) does not seem to have helped... I am not sure what I am doing wrong exactly. As you can see I have increased it by 100%, prior, I tried to increase it with 40-70% but none worked.

I think this is for the MUMPS list, but it appears you have a lot of fill-in for this matrix and it might be too much for a direct solver.

  Thanks,

    Matt

On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:


On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi I am getting an error using MUMPS.
How I run it:
bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

The error output:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-36536

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-81813

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28194

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-27685

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30519

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28435

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30110

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
#1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[15]PETSC ERROR: PETSc Option Table entries:
[15]PETSC ERROR: -ksp_error_if_not_converged
[15]PETSC ERROR: -ksp_max_it 1
[15]PETSC ERROR: -ksp_monitor_true_residual
[15]PETSC ERROR: -ksp_type richardson
[15]PETSC ERROR: -log_view
[15]PETSC ERROR: -pc_factor_mat_solver_type mumps
[15]PETSC ERROR: -pc_type lu
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
Problem of workspace allocation of size INFO(2) during the factorization or solve steps. The size that the package tried to allocate with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then the size that the package requested is obtained by multiplying the absolute value of INFO(2) by 1 million. In general, the unit for INFO(2) is the number of scalar entries of  the type of the input matrix (real, complex, single or double precision).

Not quite sure what to make of it as the first error indicates size INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
But you can see there are processes with negative INFO(2). According to MUMPS manual, you can try to increase ICNTL(14).


Thanks


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/24263b8a/attachment-0001.html>

From knepley at gmail.com  Mon Jan  7 14:52:50 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 7 Jan 2019 15:52:50 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
Message-ID: <CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>

On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com> wrote:

> thanks Matt,
>
> I posted a slightly modified example
>
> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>
> and changes from the orginal
>
> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>
>
> This is what I'm seeing
>
> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>
>
> I'm refining based on this element centroid:
>
> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>
> unrefined -
> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>
> refined -
> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>
>
> Maybe I don't understand the refinement algorithm, but the refinement is a
> little offset from what I was expecting.
>

Okay, I have that working. I will make a branch and put this in it. For
right now, I attach ex45.c (but it will not run since I put in stuff to
take lower/upper from the command line). 3D refinement just sucks, but you
can see it is refining cell 7, not 8. Its just that refinement
propagates a long, long way.

I have pretty much abandoned this type of refinement since it will not work
in parallel, and the mesh generator
interface is poor. I am transitioning everything to

  - uniform refinement (ha!)
  - cell marking (p4est likes this)
  - metric tensor (Pragmatic likes this)

I have some routines to convert marking <--> metric but they are not
perfect I think. I can try and help do what you want with
AMR if I have a better idea what it is.

  Thanks,

    Matt


>
> Also, can you pass an application context to the user refinement function ?
>
> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>
>
>
> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> Starting with TS example 45
>>>
>>> $ pwd
>>> /opt/apps/PETSc/petsc-3.10.2
>>> $ ls src/ts/examples/tutorials/ex45.c
>>> src/ts/examples/tutorials/ex45.c
>>>
>>> petsc configured with: ./config/configure.py --with-shared-libraries
>>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>>> --download-ctetgen  --download-triangle --with-debugging=yes
>>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>>> --download-pnetcdf
>>>
>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>> DMPlexSetRefinementFunction.
>>> It generally seems to be working, except that the refined element is
>>> slightly offset from what I was expecting.
>>> Based on my application specific criteria, element id number 7 is
>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>> paraview, it looks like element id number 8 is being refined. See attached
>>> pic.
>>>
>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>
>>> Is it possible that the maxVolumes array is 'off by one' when
>>> transfering to tetgen data structures somehow ?
>>>
>>
>> I looked through and cannot see it by eye. Could you send me your
>> modified example and I will walk through it with the
>> debugger?
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>
>>>
>>>
>>> (gdb) bt
>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>> dmRefined=0x7fffffffb938) at
>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>> dmRefined=0x7fffffffb938) at
>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>> dmf=0x7fffffffb938) at
>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/2e311da9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: check.png
Type: image/png
Size: 88215 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/2e311da9/attachment-0001.png>

From fuentesdt at gmail.com  Mon Jan  7 15:27:25 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Mon, 7 Jan 2019 15:27:25 -0600
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
Message-ID: <CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>

ha! thanks for you time on this Matt. I'm trying to generate a mesh from
image segmentation data.
I would like to use an image segmentation to guide the refinement. Figure
25 of this paper - https://www.ices.utexas.edu/media/reports/2017/1707.pdf



On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com> wrote:
>
>> thanks Matt,
>>
>> I posted a slightly modified example
>>
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>
>> and changes from the orginal
>>
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>
>>
>> This is what I'm seeing
>>
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>
>>
>> I'm refining based on this element centroid:
>>
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>
>> unrefined -
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>
>> refined -
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>
>>
>> Maybe I don't understand the refinement algorithm, but the refinement is
>> a little offset from what I was expecting.
>>
>
> Okay, I have that working. I will make a branch and put this in it. For
> right now, I attach ex45.c (but it will not run since I put in stuff to
> take lower/upper from the command line). 3D refinement just sucks, but you
> can see it is refining cell 7, not 8. Its just that refinement
> propagates a long, long way.
>
> I have pretty much abandoned this type of refinement since it will not
> work in parallel, and the mesh generator
> interface is poor. I am transitioning everything to
>
>   - uniform refinement (ha!)
>   - cell marking (p4est likes this)
>   - metric tensor (Pragmatic likes this)
>
> I have some routines to convert marking <--> metric but they are not
> perfect I think. I can try and help do what you want with
> AMR if I have a better idea what it is.
>
>   Thanks,
>
>     Matt
>
>
>>
>> Also, can you pass an application context to the user refinement function
>> ?
>>
>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>
>>
>>
>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi,
>>>>
>>>> Starting with TS example 45
>>>>
>>>> $ pwd
>>>> /opt/apps/PETSc/petsc-3.10.2
>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>> src/ts/examples/tutorials/ex45.c
>>>>
>>>> petsc configured with: ./config/configure.py --with-shared-libraries
>>>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>>>> --download-ctetgen  --download-triangle --with-debugging=yes
>>>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>>>> --download-pnetcdf
>>>>
>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>> DMPlexSetRefinementFunction.
>>>> It generally seems to be working, except that the refined element is
>>>> slightly offset from what I was expecting.
>>>> Based on my application specific criteria, element id number 7 is
>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>> pic.
>>>>
>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>
>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>> transfering to tetgen data structures somehow ?
>>>>
>>>
>>> I looked through and cannot see it by eye. Could you send me your
>>> modified example and I will walk through it with the
>>> debugger?
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>
>>>>
>>>>
>>>> (gdb) bt
>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>> dmRefined=0x7fffffffb938) at
>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>> dmRefined=0x7fffffffb938) at
>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>> dmf=0x7fffffffb938) at
>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190107/7672ce78/attachment.html>

From tempohoper at gmail.com  Tue Jan  8 02:58:48 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 8 Jan 2019 08:58:48 +0000
Subject: [petsc-users] Configuring PETSc with OpenMPI
In-Reply-To: <alpine.LFD.2.21.1901071038440.16192@sb>
References: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>
	<alpine.LFD.2.21.1901071038440.16192@sb>
Message-ID: <CAE3k7C5LSzJpWzsTGweH5yAfYBN_7SzVExQTgqKzucq1kB1Qog@mail.gmail.com>

Thanks Satish for quick response,

I tried both of the above, first removing the options --with-cc etc. and
then explicitly specifying the path e.g.
--with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
Netiher worked, the error is still the same telling me "did not work" I
have attached the log file.

Thanks

On Mon, Jan 7, 2019 at 4:39 PM Balay, Satish <balay at mcs.anl.gov> wrote:

> Configure Options: --configModules=PETSc.Configure
> --optionsModule=config.compilerOptions PETSC_ARCH=linux-cumulus-hpc
> --with-cc=gcc --with-fc=gfortran --with-cxx=g++
> --with-mpi-dir=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/
> --download-parmetis --download-metis --download-ptscotch
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-fblaslapack=1 --download-cmake
>
> ' --with-cc=gcc --with-fc=gfortran --with-cxx=g++' prevents usage of mpicc
> etc - so remove these options when using with-mpi-dir
>
> Or use:
>
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
>
> Satish
>
>
> On Mon, 7 Jan 2019, Sal Am via petsc-users wrote:
>
> > Added the log file.
> >
> > >From OpenMPI:
> >
> > > The only special configuration that you need to build PETSc is to
> ensure
> > > that Open MPI's wrapper compilers (i.e., mpicc and mpif77) are in your
> > > $PATH before running the PETSc configure.py script.
> > >
> > > PETSc should then automatically find Open MPI's wrapper compilers and
> > > correctly build itself using Open MPI.
> > >
> > The OpenMPI dir is on my PATH which contain mpicc and mpif77.
> >
> > This is on a HPC, if that matters.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/2dbf05eb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2109843 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/2dbf05eb/attachment-0001.obj>

From yann.jobic at univ-amu.fr  Tue Jan  8 03:31:58 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Tue, 8 Jan 2019 10:31:58 +0100
Subject: [petsc-users] basis functions of high order approximation (FE)
Message-ID: <74e50ec0-5eac-5a6b-4f6a-b3fa37a53d6f@univ-amu.fr>

Dear Petsc Users,

I've been playing with the option "space_degree", in 2D, for a space 
discretisation of 4 cells (2x2), for a poisson problem, and i wonder 
what are the underlying concepts.

With a space degree 2, i get a ?? 9x9 ? algebraic system, and i've got a 
solution convergence order of 3.

With a space degree 3, i get a 25x25 algebraic system, and i've got a 
solution convergence order of 4.

So my question is : what are the basis functions associated with each 
space degree ?

Are they lagrange polynomials or something else ?

Thanks!

Yann


From tempohoper at gmail.com  Tue Jan  8 03:37:53 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 8 Jan 2019 09:37:53 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAGCphBtyp2jD96FZGz4hAZ_TyO7VBq98bQZpZUCOkMC-uU12sg@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
	<CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
	<DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>
	<CAE3k7C4L2_vvWJEnxmByfzVE8kD4D08V0zTh6dSZuRiWxf0u=A@mail.gmail.com>
	<CAGCphBtyp2jD96FZGz4hAZ_TyO7VBq98bQZpZUCOkMC-uU12sg@mail.gmail.com>
Message-ID: <CAE3k7C4EB8RdMySGSgBBTX9kU9Shg-tbcYcknKeG8LvxzwsaMw@mail.gmail.com>

>
> How large is the matrix, its dimension and number of non-zeros?
>
Can you run it sequentially or with less number of processors?
>
The matrix is 139603x139603 and with 15975113 of non-zero elements.
I tried using the iterative solver and started on one node then increased
to two there was no performance increase in using 2 nodes with 32 cores on
each node vs using 1 node with 32 cores (both took ~12min). Increasing the
number of nodes seems to crash it though.

 Can you run your case on different machine?
>
Running it on the work machine, it seems to work using the iterative solver
but will give me similar error I first posted if I use MUMPS (INFOG(1)=-9,
INFO(2)=0)
.

On Mon, Jan 7, 2019 at 4:47 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Sal:
>
>> Recent versions of MUMPS have bugs when using SCALAPACK. You may also try
>>> to disable parallelism on the root node by using
>>
>> tried that, does not seem to help... but it is odd. this is not a big
>> system, it is small and made for testing.
>>
>
> How large is the matrix, its dimension and number of non-zeros?
> Can you run it sequentially or with less number of processors?
>
>>
>> perhaps there is something wrong with the MPI....If anyone is using the
>> PBSPro for queueing system, could it be the incompatibility between the
>> MPICH and the PBS? Should one use OpenMPI?
>>
>  Can you run your case on different machine?
> Hong
>
>>
>> On Thu, Jan 3, 2019 at 3:55 PM Stefano Zampini <stefano.zampini at gmail.com>
>> wrote:
>>
>>> Recent versions of MUMPS have bugs when using SCALAPACK. You may also
>>> try to disable parallelism on the root node by using
>>>
>>>  -mat_mumps_icntl_13 1
>>>
>>> On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>> You may try different matrix orderings, or try superlu_dist.
>>> Hong
>>>
>>>
>>> On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Error in external library
>>>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-22193
>>>>>
>>>>> [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [4]PETSC ERROR: Error in external library
>>>>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-24197
>>>>>
>>>>> [4]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [16]PETSC ERROR: Error in external library
>>>>> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [16]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [20]PETSC ERROR: Error in external library
>>>>> [20]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [20]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [24]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [24]PETSC ERROR: Error in external library
>>>>> [24]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [24]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [24]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>> [24]PETSC ERROR: #4 PCSetUp() line 932 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>> [24]PETSC ERROR: #5 KSPSetUp() line 391 in
>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [32]PETSC ERROR: Error in external library
>>>>> [32]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>
>>>>> [32]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [32]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>> [32]PETSC ERROR: #4 PCSetUp() line 932 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>> [32]PETSC ERROR: #5 KSPSetUp() line 391 in
>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [8]PETSC ERROR: Error in external library
>>>>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-35665
>>>>>
>>>>> [8]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>> [8]PETSC ERROR: #4 PCSetUp() line 932 in
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>> [8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [12]PETSC ERROR: Error in external library
>>>>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFOG(1)=-13, INFO(2)=-16661
>>>>>
>>>>> [12]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu
>>>>> Jan  3 04:32:43 2019
>>>>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>> [12]PETSC ERROR: #6 main() line 68 in
>>>>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>>>>> [24]PETSC ERROR: PETSc Option Table entries:
>>>>> [24]PETSC ERROR: -ksp_error_if_not_converged
>>>>> [24]PETSC ERROR: -ksp_max_it 1
>>>>> [24]PETSC ERROR: -ksp_monitor_true_residual
>>>>> [24]PETSC ERROR: -ksp_type richardson
>>>>> [24]PETSC ERROR: -log_view
>>>>> [24]PETSC ERROR: -mat_mumps_icntl_14 100
>>>>> [24]PETSC ERROR: -pc_factor_mat_solver_type mumps
>>>>> [24]PETSC ERROR: -pc_type lu
>>>>> [24]PETSC ERROR: ----------------End of Error Message -------send
>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>
>>>>> Sorry to bother again, but increasing ICNTL(14) does not seem to have
>>>>> helped... I am not sure what I am doing wrong exactly. As you can see I
>>>>> have increased it by 100%, prior, I tried to increase it with 40-70% but
>>>>> none worked.
>>>>>
>>>>
>>>> I think this is for the MUMPS list, but it appears you have a lot of
>>>> fill-in for this matrix and it might be too much for a direct solver.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi I am getting an error using MUMPS.
>>>>>>> How I run it:
>>>>>>> bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu
>>>>>>> -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual
>>>>>>> -log_view -ksp_error_if_not_converged
>>>>>>>
>>>>>>> The error output:
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Error in external library
>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [1]PETSC ERROR: Error in external library
>>>>>>> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [1]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [6]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [6]PETSC ERROR: Error in external library
>>>>>>> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [8]PETSC ERROR: Error in external library
>>>>>>> [8]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [8]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR:
>>>>>>> --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [15]PETSC ERROR: Error in external library
>>>>>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-36536
>>>>>>>
>>>>>>> [15]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [15]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-81813
>>>>>>>
>>>>>>> [0]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: Error in external library
>>>>>>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [2]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [3]PETSC ERROR: Error in external library
>>>>>>> [3]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-28194
>>>>>>>
>>>>>>> [3]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [4]PETSC ERROR: Error in external library
>>>>>>> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [4]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [5]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [5]PETSC ERROR: Error in external library
>>>>>>> [5]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-27685
>>>>>>>
>>>>>>> [5]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [5]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [7]PETSC ERROR: Error in external library
>>>>>>> [7]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [7]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>> ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12
>>>>>>> 10:44:13 2018
>>>>>>> [8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc
>>>>>>> --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>> [8]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>>>> [10]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [10]PETSC ERROR: Error in external library
>>>>>>> [10]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [10]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error
>>>>>>> Message --------------------------------------------------------------
>>>>>>> [11]PETSC ERROR: Error in external library
>>>>>>> [11]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-30519
>>>>>>>
>>>>>>> [11]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error
>>>>>>> Message --------------------------------------------------------------
>>>>>>> [12]PETSC ERROR: Error in external library
>>>>>>> [12]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-28435
>>>>>>>
>>>>>>> [12]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error
>>>>>>> Message --------------------------------------------------------------
>>>>>>> [13]PETSC ERROR: Error in external library
>>>>>>> [13]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=-30110
>>>>>>>
>>>>>>> [13]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error
>>>>>>> Message --------------------------------------------------------------
>>>>>>> [14]PETSC ERROR: Error in external library
>>>>>>> [14]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFOG(1)=-13, INFO(2)=0
>>>>>>>
>>>>>>> [14]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002
>>>>>>> Wed Dec 12 10:44:13 2018
>>>>>>> [14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt
>>>>>>> --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=no --download-scalapack --download-superlu
>>>>>>> --download-mpich --download-fblaslapack=1 --download-cmake
>>>>>>> [14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>> #1 MatFactorNumeric_MUMPS() line 1230 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
>>>>>>> [15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in
>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>> [15]PETSC ERROR: #3 PCSetUp_LU() line 131 in
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
>>>>>>> [15]PETSC ERROR: #4 PCSetUp() line 932 in
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>> [15]PETSC ERROR: #5 KSPSetUp() line 391 in
>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>> [15]PETSC ERROR: #6 main() line 68 in
>>>>>>> /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
>>>>>>> [15]PETSC ERROR: PETSc Option Table entries:
>>>>>>> [15]PETSC ERROR: -ksp_error_if_not_converged
>>>>>>> [15]PETSC ERROR: -ksp_max_it 1
>>>>>>> [15]PETSC ERROR: -ksp_monitor_true_residual
>>>>>>> [15]PETSC ERROR: -ksp_type richardson
>>>>>>> [15]PETSC ERROR: -log_view
>>>>>>> [15]PETSC ERROR: -pc_factor_mat_solver_type mumps
>>>>>>> [15]PETSC ERROR: -pc_type lu
>>>>>>> [15]PETSC ERROR: ----------------End of Error Message -------send
>>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>>>
>>>>>>> From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
>>>>>>> Problem of workspace allocation of size INFO(2) during the
>>>>>>> factorization or solve steps. The size that the package tried to allocate
>>>>>>> with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is
>>>>>>> negative, then the size that the package requested is obtained by
>>>>>>> multiplying the absolute value of INFO(2) by 1 million. In general, the
>>>>>>> unit for INFO(2) is the number of scalar entries of  the type of the input
>>>>>>> matrix (real, complex, single or double precision).
>>>>>>>
>>>>>>> Not quite sure what to make of it as the first error indicates size
>>>>>>> INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
>>>>>>>
>>>>>> But you can see there are processes with negative INFO(2). According
>>>>>> to MUMPS manual, you can try to increase ICNTL(14).
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/a9ec2130/attachment-0001.html>

From balay at mcs.anl.gov  Tue Jan  8 08:07:52 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Tue, 8 Jan 2019 14:07:52 +0000
Subject: [petsc-users] Configuring PETSc with OpenMPI
In-Reply-To: <CAE3k7C5LSzJpWzsTGweH5yAfYBN_7SzVExQTgqKzucq1kB1Qog@mail.gmail.com>
References: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>
	<alpine.LFD.2.21.1901071038440.16192@sb>
	<CAE3k7C5LSzJpWzsTGweH5yAfYBN_7SzVExQTgqKzucq1kB1Qog@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901080806240.27321@sb>

Should have clarified: when using --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc - removing -with-mpi-dir option.

i.e - try:

/configure PETSC_ARCH=linux-cumulus-hpc --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake

Satish

On Tue, 8 Jan 2019, Sal Am via petsc-users wrote:

> Thanks Satish for quick response,
> 
> I tried both of the above, first removing the options --with-cc etc. and
> then explicitly specifying the path e.g.
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
> Netiher worked, the error is still the same telling me "did not work" I
> have attached the log file.
> 
> Thanks
> 
> On Mon, Jan 7, 2019 at 4:39 PM Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> > Configure Options: --configModules=PETSc.Configure
> > --optionsModule=config.compilerOptions PETSC_ARCH=linux-cumulus-hpc
> > --with-cc=gcc --with-fc=gfortran --with-cxx=g++
> > --with-mpi-dir=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/
> > --download-parmetis --download-metis --download-ptscotch
> > --download-superlu_dist --download-mumps --with-scalar-type=complex
> > --with-debugging=no --download-scalapack --download-superlu
> > --download-fblaslapack=1 --download-cmake
> >
> > ' --with-cc=gcc --with-fc=gfortran --with-cxx=g++' prevents usage of mpicc
> > etc - so remove these options when using with-mpi-dir
> >
> > Or use:
> >
> > --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
> >
> > Satish
> >
> >
> > On Mon, 7 Jan 2019, Sal Am via petsc-users wrote:
> >
> > > Added the log file.
> > >
> > > >From OpenMPI:
> > >
> > > > The only special configuration that you need to build PETSc is to
> > ensure
> > > > that Open MPI's wrapper compilers (i.e., mpicc and mpif77) are in your
> > > > $PATH before running the PETSc configure.py script.
> > > >
> > > > PETSc should then automatically find Open MPI's wrapper compilers and
> > > > correctly build itself using Open MPI.
> > > >
> > > The OpenMPI dir is on my PATH which contain mpicc and mpif77.
> > >
> > > This is on a HPC, if that matters.
> > >
> >
> >
> 


From hzhang at mcs.anl.gov  Tue Jan  8 10:13:39 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 8 Jan 2019 16:13:39 +0000
Subject: [petsc-users] MUMPS Error
In-Reply-To: <CAE3k7C4EB8RdMySGSgBBTX9kU9Shg-tbcYcknKeG8LvxzwsaMw@mail.gmail.com>
References: <CAE3k7C4QtUhq0kU1J+Kp+w3uirgTp4H1oXo_VZoJRgF8byBYVg@mail.gmail.com>
	<CA+MQGp9_mfzJdtTW0AEGVBPd39eCCmjVJytjzR382NZXacj41Q@mail.gmail.com>
	<CAE3k7C4vcNo6iLy855j_qc1SR+jd+KhYod4xWTp=mufw48jO6Q@mail.gmail.com>
	<CAMYG4GnKMtuvOV+N0e0fC0EcZDrnvRy4Hk04gni3ZhfwsiS0WQ@mail.gmail.com>
	<CAGCphBt-H4jdsjhJdeTU--BvGD_OyAE3Oiig4Qnrtj5GFRAOMQ@mail.gmail.com>
	<DE63B898-0775-4F18-A341-5069A3AF3857@gmail.com>
	<CAE3k7C4L2_vvWJEnxmByfzVE8kD4D08V0zTh6dSZuRiWxf0u=A@mail.gmail.com>
	<CAGCphBtyp2jD96FZGz4hAZ_TyO7VBq98bQZpZUCOkMC-uU12sg@mail.gmail.com>
	<CAE3k7C4EB8RdMySGSgBBTX9kU9Shg-tbcYcknKeG8LvxzwsaMw@mail.gmail.com>
Message-ID: <CAGCphBv+_Utdj3cL41U1iN1ay160noHze0=1YiT3yBdrGjtM_Q@mail.gmail.com>

Sal:
How large is the matrix, its dimension and number of non-zeros?
Can you run it sequentially or with less number of processors?
The matrix is 139603x139603 and with 15975113 of non-zero elements.
This is quite large for direct solver, but it is very sparse. Have you tried different matrix orderings to reduce fillings?
You may try to use superu_dist which might consume less memory than mumps.

I tried using the iterative solver and started on one node then increased to two there was no performance increase in using 2 nodes with 32 cores on each node vs using 1 node with 32 cores (both took ~12min). Increasing the number of nodes seems to crash it though.
What  iterative solver do you use? Increasing the number of nodes likely changes preconditioner.

 Can you run your case on different machine?
Running it on the work machine, it seems to work using the iterative solver but will give me similar error I first posted if I use MUMPS (INFOG(1)=-9, INFO(2)=0)
We need know what   iterative solver is being used. Run your code with '-ksp_view'.

Hong
.

On Mon, Jan 7, 2019 at 4:47 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Sal:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using
tried that, does not seem to help... but it is odd. this is not a big system, it is small and made for testing.

How large is the matrix, its dimension and number of non-zeros?
Can you run it sequentially or with less number of processors?

perhaps there is something wrong with the MPI....If anyone is using the PBSPro for queueing system, could it be the incompatibility between the MPICH and the PBS? Should one use OpenMPI?
 Can you run your case on different machine?
Hong

On Thu, Jan 3, 2019 at 3:55 PM Stefano Zampini <stefano.zampini at gmail.com<mailto:stefano.zampini at gmail.com>> wrote:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using

 -mat_mumps_icntl_13 1

On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

You may try different matrix orderings, or try superlu_dist.
Hong


On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-22193

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-24197

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[24]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-35665

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[8]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-16661

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -ksp_error_if_not_converged
[24]PETSC ERROR: -ksp_max_it 1
[24]PETSC ERROR: -ksp_monitor_true_residual
[24]PETSC ERROR: -ksp_type richardson
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_mumps_icntl_14 100
[24]PETSC ERROR: -pc_factor_mat_solver_type mumps
[24]PETSC ERROR: -pc_type lu
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

Sorry to bother again, but increasing ICNTL(14) does not seem to have helped... I am not sure what I am doing wrong exactly. As you can see I have increased it by 100%, prior, I tried to increase it with 40-70% but none worked.

I think this is for the MUMPS list, but it appears you have a lot of fill-in for this matrix and it might be too much for a direct solver.

  Thanks,

    Matt

On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:


On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi I am getting an error using MUMPS.
How I run it:
bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

The error output:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-36536

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-81813

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28194

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-27685

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30519

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28435

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30110

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
#1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[15]PETSC ERROR: PETSc Option Table entries:
[15]PETSC ERROR: -ksp_error_if_not_converged
[15]PETSC ERROR: -ksp_max_it 1
[15]PETSC ERROR: -ksp_monitor_true_residual
[15]PETSC ERROR: -ksp_type richardson
[15]PETSC ERROR: -log_view
[15]PETSC ERROR: -pc_factor_mat_solver_type mumps
[15]PETSC ERROR: -pc_type lu
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
Problem of workspace allocation of size INFO(2) during the factorization or solve steps. The size that the package tried to allocate with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then the size that the package requested is obtained by multiplying the absolute value of INFO(2) by 1 million. In general, the unit for INFO(2) is the number of scalar entries of  the type of the input matrix (real, complex, single or double precision).

Not quite sure what to make of it as the first error indicates size INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
But you can see there are processes with negative INFO(2). According to MUMPS manual, you can try to increase ICNTL(14).


Thanks


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/a5a8cba3/attachment-0001.html>

From jed at jedbrown.org  Tue Jan  8 10:30:15 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 08 Jan 2019 09:30:15 -0700
Subject: [petsc-users] basis functions of high order approximation (FE)
In-Reply-To: <74e50ec0-5eac-5a6b-4f6a-b3fa37a53d6f@univ-amu.fr>
References: <74e50ec0-5eac-5a6b-4f6a-b3fa37a53d6f@univ-amu.fr>
Message-ID: <87a7kbnmco.fsf@jedbrown.org>

Yes, tensor product of Lagrange polynomials, often written as Q_k.
Degrees of freedom associated with Dirichlet boundary conditions have
been removed in the systems you're looking at.

Yann Jobic via petsc-users <petsc-users at mcs.anl.gov> writes:

> Dear Petsc Users,
>
> I've been playing with the option "space_degree", in 2D, for a space 
> discretisation of 4 cells (2x2), for a poisson problem, and i wonder 
> what are the underlying concepts.
>
> With a space degree 2, i get a ?? 9x9 ? algebraic system, and i've got a 
> solution convergence order of 3.
>
> With a space degree 3, i get a 25x25 algebraic system, and i've got a 
> solution convergence order of 4.
>
> So my question is : what are the basis functions associated with each 
> space degree ?
>
> Are they lagrange polynomials or something else ?
>
> Thanks!
>
> Yann

From knepley at gmail.com  Tue Jan  8 11:19:58 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Jan 2019 12:19:58 -0500
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
Message-ID: <CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>

On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com> wrote:

> ha! thanks for you time on this Matt. I'm trying to generate a mesh from
> image segmentation data.
> I would like to use an image segmentation to guide the refinement. Figure
> 25 of this paper - https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>

Very cool. We can do that. Lets make an example and iterate. Want to make a
repo you control? I will
stick in a PETSc example that refines meshes (maybe Plex ex19), and we can
experiment with both p4est
and Pragmatic (installation of those is hardest part :) This sounds great.

   Matt


> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>> wrote:
>>
>>> thanks Matt,
>>>
>>> I posted a slightly modified example
>>>
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>
>>> and changes from the orginal
>>>
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>
>>>
>>> This is what I'm seeing
>>>
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>
>>>
>>> I'm refining based on this element centroid:
>>>
>>>
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>
>>> unrefined -
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>
>>> refined -
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>
>>>
>>> Maybe I don't understand the refinement algorithm, but the refinement is
>>> a little offset from what I was expecting.
>>>
>>
>> Okay, I have that working. I will make a branch and put this in it. For
>> right now, I attach ex45.c (but it will not run since I put in stuff to
>> take lower/upper from the command line). 3D refinement just sucks, but you
>> can see it is refining cell 7, not 8. Its just that refinement
>> propagates a long, long way.
>>
>> I have pretty much abandoned this type of refinement since it will not
>> work in parallel, and the mesh generator
>> interface is poor. I am transitioning everything to
>>
>>   - uniform refinement (ha!)
>>   - cell marking (p4est likes this)
>>   - metric tensor (Pragmatic likes this)
>>
>> I have some routines to convert marking <--> metric but they are not
>> perfect I think. I can try and help do what you want with
>> AMR if I have a better idea what it is.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>>
>>> Also, can you pass an application context to the user refinement
>>> function ?
>>>
>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>
>>>
>>>
>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Starting with TS example 45
>>>>>
>>>>> $ pwd
>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>> src/ts/examples/tutorials/ex45.c
>>>>>
>>>>> petsc configured with: ./config/configure.py --with-shared-libraries
>>>>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>>>>> --download-ctetgen  --download-triangle --with-debugging=yes
>>>>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>>>>> --download-pnetcdf
>>>>>
>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>> DMPlexSetRefinementFunction.
>>>>> It generally seems to be working, except that the refined element is
>>>>> slightly offset from what I was expecting.
>>>>> Based on my application specific criteria, element id number 7 is
>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>> pic.
>>>>>
>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>
>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>> transfering to tetgen data structures somehow ?
>>>>>
>>>>
>>>> I looked through and cannot see it by eye. Could you send me your
>>>> modified example and I will walk through it with the
>>>> debugger?
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>>
>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>
>>>>>
>>>>>
>>>>> (gdb) bt
>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>> dmRefined=0x7fffffffb938) at
>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>> dmRefined=0x7fffffffb938) at
>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>> dmf=0x7fffffffb938) at
>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/6fe9d9bf/attachment.html>

From fuentesdt at gmail.com  Tue Jan  8 11:29:34 2019
From: fuentesdt at gmail.com (David Fuentes)
Date: Tue, 8 Jan 2019 11:29:34 -0600
Subject: [petsc-users] DMPlexSetRefinementFunction
In-Reply-To: <CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
References: <CAKnkmZy3Gsxy3WuqjS=q3P2-7sQiLmziPLkyUZ+H1KnmkYUwFQ@mail.gmail.com>
	<CAMYG4GmBKdJKbMMP4Sf5tZv6+-sXrbiJ_t6_GyrcgL55h10scA@mail.gmail.com>
	<CAKnkmZycPfDquHX8ZtndgwU1JKCRL2ahgQd4xWShyxGnNaycWw@mail.gmail.com>
	<CAMYG4GmOLemviVwdUWKQLdD8oResos8nB7mUcQWf_teP0q_+sg@mail.gmail.com>
	<CAKnkmZx=4kAwy_Pb8RzSXesvibj6baECQTWM8OPQxs9WNJtZyQ@mail.gmail.com>
	<CAMYG4GmArTbUeDZJLLyVX4k+nJFzWg2419S7G-_gATz-F8HN0g@mail.gmail.com>
Message-ID: <CAKnkmZx9inzaDYLo=+xaRBeKxoAJYEF21omq9gb1F3cbZB-hjQ@mail.gmail.com>

sounds great! i've been working out of this guy -
https://github.com/fuentesdt/thermoembo

and loading vtk data for the segmentation:
https://github.com/fuentesdt/thermoembo/blob/master/tutorials/exac.c#L50

I can clean this up and separate into relevant directories ? or start an
new repo? which ever you prefer.



On Tue, Jan 8, 2019 at 11:20 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Jan 7, 2019 at 4:27 PM David Fuentes <fuentesdt at gmail.com> wrote:
>
>> ha! thanks for you time on this Matt. I'm trying to generate a mesh from
>> image segmentation data.
>> I would like to use an image segmentation to guide the refinement. Figure
>> 25 of this paper -
>> https://www.ices.utexas.edu/media/reports/2017/1707.pdf
>>
>
> Very cool. We can do that. Lets make an example and iterate. Want to make
> a repo you control? I will
> stick in a PETSc example that refines meshes (maybe Plex ex19), and we can
> experiment with both p4est
> and Pragmatic (installation of those is hardest part :) This sounds great.
>
>    Matt
>
>
>> On Mon, Jan 7, 2019 at 2:53 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Mon, Jan 7, 2019 at 11:41 AM David Fuentes <fuentesdt at gmail.com>
>>> wrote:
>>>
>>>> thanks Matt,
>>>>
>>>> I posted a slightly modified example
>>>>
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c
>>>>
>>>> and changes from the orginal
>>>>
>>>>
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refinement.diff
>>>>
>>>>
>>>> This is what I'm seeing
>>>>
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/paraview.png
>>>>
>>>>
>>>> I'm refining based on this element centroid:
>>>>
>>>>
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/testcentroid.stl
>>>>
>>>> unrefined -
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/unrefined.vtu
>>>>
>>>> refined -
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/refined.vtu
>>>>
>>>>
>>>> Maybe I don't understand the refinement algorithm, but the refinement
>>>> is a little offset from what I was expecting.
>>>>
>>>
>>> Okay, I have that working. I will make a branch and put this in it. For
>>> right now, I attach ex45.c (but it will not run since I put in stuff to
>>> take lower/upper from the command line). 3D refinement just sucks, but you
>>> can see it is refining cell 7, not 8. Its just that refinement
>>> propagates a long, long way.
>>>
>>> I have pretty much abandoned this type of refinement since it will not
>>> work in parallel, and the mesh generator
>>> interface is poor. I am transitioning everything to
>>>
>>>   - uniform refinement (ha!)
>>>   - cell marking (p4est likes this)
>>>   - metric tensor (Pragmatic likes this)
>>>
>>> I have some routines to convert marking <--> metric but they are not
>>> perfect I think. I can try and help do what you want with
>>> AMR if I have a better idea what it is.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>>
>>>> Also, can you pass an application context to the user refinement
>>>> function ?
>>>>
>>>> https://github.com/fuentesdt/dmplexrefinement/blob/master/ex45.c#L119
>>>>
>>>>
>>>>
>>>> On Thu, Jan 3, 2019 at 7:49 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Jan 2, 2019 at 7:28 PM David Fuentes via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Starting with TS example 45
>>>>>>
>>>>>> $ pwd
>>>>>> /opt/apps/PETSc/petsc-3.10.2
>>>>>> $ ls src/ts/examples/tutorials/ex45.c
>>>>>> src/ts/examples/tutorials/ex45.c
>>>>>>
>>>>>> petsc configured with: ./config/configure.py --with-shared-libraries
>>>>>> --with-clanguage=c++ --CFLAGS='-g -O0' --CXXFLAGS='-g -O0'  --with-c2html=0
>>>>>> --download-ctetgen  --download-triangle --with-debugging=yes
>>>>>> --download-netcdf  --download-zlib  --download-exodusii --download-hdf5
>>>>>> --download-pnetcdf
>>>>>>
>>>>>> I'm trying to refine the DMPlexCreateBoxMesh  with the
>>>>>> DMPlexSetRefinementFunction.
>>>>>> It generally seems to be working, except that the refined element is
>>>>>> slightly offset from what I was expecting.
>>>>>> Based on my application specific criteria, element id number 7 is
>>>>>> flagged to be refined by the DMPlexSetRefinementFunction but when I open in
>>>>>> paraview, it looks like element id number 8 is being refined. See attached
>>>>>> pic.
>>>>>>
>>>>>> [image: Screen Shot 2019-01-02 at 6.02.02 PM.png]
>>>>>> [image: Screen Shot 2019-01-02 at 6.02.11 PM.png]
>>>>>>
>>>>>> Is it possible that the maxVolumes array is 'off by one' when
>>>>>> transfering to tetgen data structures somehow ?
>>>>>>
>>>>>
>>>>> I looked through and cannot see it by eye. Could you send me your
>>>>> modified example and I will walk through it with the
>>>>> debugger?
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>>
>>>>>> https://bitbucket.org/petsc/petsc/src/bd27d3f284687498e4c4678d234c0e308a5bc236/src/dm/impls/plex/plexadapt.c?at=master&fileviewer=file-view-default#plexadapt.c-252
>>>>>>
>>>>>>
>>>>>>
>>>>>> (gdb) bt
>>>>>> #0  DMPlexRefine_CTetgen (dm=0x932180, maxVolumes=0x919710,
>>>>>> dmRefined=0x7fffffffb938) at
>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/generators/ctetgen/ctetgenerate.c:182
>>>>>> #1  0x00007ffff6b76401 in DMPlexRefine_Internal (dm=0x932180,
>>>>>> adaptLabel=0x0, dmRefined=0x7fffffffb938) at
>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexadapt.c:252
>>>>>> #2  0x00007ffff6b72720 in DMRefine_Plex (dm=0x932180, comm=0x6b,
>>>>>> dmRefined=0x7fffffffb938) at
>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/impls/plex/plexrefine.c:10361
>>>>>> #3  0x00007ffff6dad8ff in DMRefine (dm=0x932180, comm=0x6b,
>>>>>> dmf=0x7fffffffb938) at
>>>>>> /opt/apps/PETSc/petsc-3.10.2/src/dm/interface/dm.c:1808
>>>>>> #4  0x0000000000405274 in CreateMesh (comm=0x7ffff5891680
>>>>>> <ompi_mpi_comm_world>, dm=0x7fffffffb9d0, ctx=0x7fffffffba00) at
>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:253
>>>>>> #5  0x00000000004063c4 in main (argc=32, argv=0x7fffffffdb68) at
>>>>>> /rsrch1/ip/dtfuentes/github/thermoembo/tutorials/exac.c:336
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/427e5881/attachment-0001.html>

From yann.jobic at univ-amu.fr  Tue Jan  8 12:45:44 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Tue, 8 Jan 2019 19:45:44 +0100
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
	<CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
	<498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>
	<CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>
Message-ID: <3c4efe88-60e6-ff3b-9267-961cba5cbdb4@univ-amu.fr>


On 05/01/2019 13:23, Matthew Knepley wrote:
> On Sat, Jan 5, 2019 at 4:04 AM Yann Jobic <yann.jobic at univ-amu.fr 
> <mailto:yann.jobic at univ-amu.fr>> wrote:
>
>
>     On 05/01/2019 02:36, Matthew Knepley wrote:
>>     On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users
>>     <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>
>>         Dear Petsc Users,
>>
>>         I'm using DMPlexCreateFromCellList to create my DM. I would
>>         like to have
>>         an order 2 geometry.
>>
>>         It's working fine in 2D for elements of type Q9.
>>
>>
>>     I do not see how that is possible. The topology does not work
>>     that way, and there
>>     is no way a 9 vertex quad was interpolated, so something else is
>>     happening.
>>
>>     Moreover, anything you input still has affine geometry. Toby has
>>     been working on
>>     non-affine geometry, and all the code is there, but right now you
>>     have to build and
>>     populate the coordinate space by hand. What this means is that
>>     you create a
>>     quadratic space for the coordinate DM (rather than the default
>>     linear), get the
>>     global coordinate vector out, and stick in the Q9 coordinates.
>>     The topology is still
>>     the same no matter what coordinates you read in.
>
>     I get the idea. So at the beginning i start with the Q4/H8 dmplex,
>     then i fill from the global coordinate vector the missing
>     coordinates, and attach this new coordinate vector to the DM.
>
> Yes.

So far, i can change the PetscSpace order, add this space to a PetscFE, 
and finally attach this PetscFE to a PetscDS (discetization) to my DM.

But if i'm doing that, the middle nodes will already be added (i 
couldn't find this function so far, could you point me the name ?). I 
can still move those middle vertex to my geometry.

But i couldn't find a way to define the quadratic space for the 
coordinate DM before all of that, at the DM (or DMPlex) level.

What way can you advise me to use in order to do that ?

Thank you,

Yann

>     Then the interpolation will be correct ?
>
> This is the tricky part. What you likely have to do is:
>
> ? 1) Build the topology
> ? 2) Run over the cells
> ? 3) For each cell, put in the coordinates using 
> DMPlexVecSetClosure(). This is redundant, but the only way
> ? ? ? you can connect with the Q9 format.
>
>     Can i use DMproject and all the petsc functions ?
>
> This is the idea. The amount of testing is so far almost nothing.
>
>     In order to output the results, i've got to write my own vtk
>     interface no ? (by splitting the Q9/H27 in 4xQ4 or 8xH8).
>
> That is a good question. Right now, it will just output Q4. However, 
> compared to everything else, writing Q9 should
> be comparatively easy.
>
>     Or is it working with the hdf5 one ?
>
> I would probably write code to to Xdmf through HDF5 rather than 
> straight VTK, but we will end up having both.
>
>     Does the p4est interface work with this setup ?
>
> It will. I do not know if it currently does. We have to check.
>
> ? Thanks,
>
> ? ? Matt
>
>>
>>     We will get around to making an interface for this soon.
>
>     Great !
>
>     Thanks,
>
>     Yann
>
>>
>>     ? Thanks,
>>
>>     ? ? ?Matt
>>
>>         I checked that it's working correctly by using
>>         DMPlexComputeCellGeometryFEM, and compute the value of the
>>         determinant
>>         (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>>
>>         I can import H8 elements, it's working fine.
>>
>>         But for H27 element, it's not working. I really don't know
>>         how to order
>>         my vertex in the cells array. So far, the determinant is zero...
>>
>>         I don't know where to look in order to find this information.
>>         I tried
>>         the function DMPlexGetRawFaces_Internal of the file
>>
>>         https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>>
>>         but it didn't help me.
>>
>>         Could you please point me where to look ?
>>
>>         Thanks, and happy new year !
>>
>>         Yann
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/dc41e6dc/attachment.html>

From yann.jobic at univ-amu.fr  Tue Jan  8 12:47:01 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Tue, 8 Jan 2019 19:47:01 +0100
Subject: [petsc-users] basis functions of high order approximation (FE)
In-Reply-To: <87a7kbnmco.fsf@jedbrown.org>
References: <74e50ec0-5eac-5a6b-4f6a-b3fa37a53d6f@univ-amu.fr>
	<87a7kbnmco.fsf@jedbrown.org>
Message-ID: <7e3b9122-a5df-0390-299e-f23ec56cdee5@univ-amu.fr>

It's clear now !

Many thanks for the explanations.

Yann

On 08/01/2019 17:30, Jed Brown wrote:
> Yes, tensor product of Lagrange polynomials, often written as Q_k.
> Degrees of freedom associated with Dirichlet boundary conditions have
> been removed in the systems you're looking at.
>
> Yann Jobic via petsc-users <petsc-users at mcs.anl.gov> writes:
>
>> Dear Petsc Users,
>>
>> I've been playing with the option "space_degree", in 2D, for a space
>> discretisation of 4 cells (2x2), for a poisson problem, and i wonder
>> what are the underlying concepts.
>>
>> With a space degree 2, i get a ?? 9x9 ? algebraic system, and i've got a
>> solution convergence order of 3.
>>
>> With a space degree 3, i get a 25x25 algebraic system, and i've got a
>> solution convergence order of 4.
>>
>> So my question is : what are the basis functions associated with each
>> space degree ?
>>
>> Are they lagrange polynomials or something else ?
>>
>> Thanks!
>>
>> Yann

From mattalo.k at gmail.com  Tue Jan  8 13:25:57 2019
From: mattalo.k at gmail.com (Kevin Mattalo)
Date: Tue, 8 Jan 2019 14:25:57 -0500
Subject: [petsc-users] Modifying Jacobian and Preconditioner in a Loop
Message-ID: <CAPiecX1dYtbriHmDd+BpBWhtUVAn-BYS-9+e7x9n68LKhitMHw@mail.gmail.com>

Hello,

I am currently solving a large linear system using a discontinuous Galerkin
approach with an inexact Newton method. I begin with a small time step and
then ramp up the time step every m outer iterations. I lag the update of
both by a certain integer multiple to improve efficiency. Because of this,
my Jacobian and therefore the preconditioner need to be modified,
reassembled, and updated in a loop. Right now, for each update I zero the
entries in my Jacobian, compute the entries, assemble the matrix, reset my
operators using KSPSetOperators() and then call KSPGetPC and KSPSetUp() to
get the new preconditioner.

Is there a more efficient way to continuously update the Jacobian?

Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/d16b8ffb/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan  8 15:03:28 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 8 Jan 2019 21:03:28 +0000
Subject: [petsc-users] Modifying Jacobian and Preconditioner in a Loop
In-Reply-To: <CAPiecX1dYtbriHmDd+BpBWhtUVAn-BYS-9+e7x9n68LKhitMHw@mail.gmail.com>
References: <CAPiecX1dYtbriHmDd+BpBWhtUVAn-BYS-9+e7x9n68LKhitMHw@mail.gmail.com>
Message-ID: <BFC03AD7-757A-4448-8A73-6A6CFA1CD633@anl.gov>



> On Jan 8, 2019, at 1:25 PM, Kevin Mattalo via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
> 
> I am currently solving a large linear system using a discontinuous Galerkin approach with an inexact Newton method. I begin with a small time step and then ramp up the time step every m outer iterations. I lag the update of both by a certain integer multiple to improve efficiency. Because of this, my Jacobian and therefore the preconditioner need to be modified, reassembled, and updated in a loop. Right now, for each update I zero the entries in my Jacobian, compute the entries, assemble the matrix, reset my operators using KSPSetOperators() and then call KSPGetPC and KSPSetUp() to get the new preconditioner.

   You don't actually need to call KSPSetOperators(), or KSPGetPC(), or KSPSetUp(). PETSc tracks if anything has changed in the matrix and then (and only then) does it rebuild the preconditioner on the next KSPSolve() call.

   Barry

> 
> Is there a more efficient way to continuously update the Jacobian?
> 
> Kevin


From tempohoper at gmail.com  Tue Jan  8 15:04:23 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 8 Jan 2019 21:04:23 +0000
Subject: [petsc-users] Configuring PETSc with OpenMPI
In-Reply-To: <alpine.LFD.2.21.1901080806240.27321@sb>
References: <CAE3k7C6wVhr9HBiUgZX8vXtn9mzA8NRrF1nKQRyWzRV07gXfrg@mail.gmail.com>
	<alpine.LFD.2.21.1901071038440.16192@sb>
	<CAE3k7C5LSzJpWzsTGweH5yAfYBN_7SzVExQTgqKzucq1kB1Qog@mail.gmail.com>
	<alpine.LFD.2.21.1901080806240.27321@sb>
Message-ID: <CAE3k7C6wn3fWe0yUfCnzKY86SZZthjjJDoBZVy1P4DVH2KvSvA@mail.gmail.com>

Thank you that worked flawlessly!

Am Di., 8. Jan. 2019, 14:07 hat Balay, Satish <balay at mcs.anl.gov>
geschrieben:

> Should have clarified: when using
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc - removing
> -with-mpi-dir option.
>
> i.e - try:
>
> /configure PETSC_ARCH=linux-cumulus-hpc
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
> --download-parmetis --download-metis --download-ptscotch
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=no --download-scalapack --download-superlu
> --download-fblaslapack=1 --download-cmake
>
> Satish
>
> On Tue, 8 Jan 2019, Sal Am via petsc-users wrote:
>
> > Thanks Satish for quick response,
> >
> > I tried both of the above, first removing the options --with-cc etc. and
> > then explicitly specifying the path e.g.
> > --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
> > Netiher worked, the error is still the same telling me "did not work" I
> > have attached the log file.
> >
> > Thanks
> >
> > On Mon, Jan 7, 2019 at 4:39 PM Balay, Satish <balay at mcs.anl.gov> wrote:
> >
> > > Configure Options: --configModules=PETSc.Configure
> > > --optionsModule=config.compilerOptions PETSC_ARCH=linux-cumulus-hpc
> > > --with-cc=gcc --with-fc=gfortran --with-cxx=g++
> > > --with-mpi-dir=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/
> > > --download-parmetis --download-metis --download-ptscotch
> > > --download-superlu_dist --download-mumps --with-scalar-type=complex
> > > --with-debugging=no --download-scalapack --download-superlu
> > > --download-fblaslapack=1 --download-cmake
> > >
> > > ' --with-cc=gcc --with-fc=gfortran --with-cxx=g++' prevents usage of
> mpicc
> > > etc - so remove these options when using with-mpi-dir
> > >
> > > Or use:
> > >
> > > --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc etc..
> > >
> > > Satish
> > >
> > >
> > > On Mon, 7 Jan 2019, Sal Am via petsc-users wrote:
> > >
> > > > Added the log file.
> > > >
> > > > >From OpenMPI:
> > > >
> > > > > The only special configuration that you need to build PETSc is to
> > > ensure
> > > > > that Open MPI's wrapper compilers (i.e., mpicc and mpif77) are in
> your
> > > > > $PATH before running the PETSc configure.py script.
> > > > >
> > > > > PETSc should then automatically find Open MPI's wrapper compilers
> and
> > > > > correctly build itself using Open MPI.
> > > > >
> > > > The OpenMPI dir is on my PATH which contain mpicc and mpif77.
> > > >
> > > > This is on a HPC, if that matters.
> > > >
> > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/8774ef1d/attachment.html>

From fdkong.jd at gmail.com  Tue Jan  8 16:11:43 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 8 Jan 2019 15:11:43 -0700
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
Message-ID: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>

Hi All,

I was wondering how to convert a regular (2D DMDA) grid coordinate (i,j) to
a global index that is used for global matrices and vectors. Do we have any
API on this?

Thanks,

Fande Kong,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/e9f61b64/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan  8 16:29:34 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 8 Jan 2019 22:29:34 +0000
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
Message-ID: <A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>



> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi All,
> 
> I was wondering how to convert a regular (2D DMDA) grid coordinate (i,j) to a global index that is used for global matrices and vectors. Do we have any API on this? 

    You can convert the global i,j to a local I,J by subtracting off the xs, ys, then convert to the local vector numbering with something like I+J*xm then add that value to the rstart returned from VecGetOwnershipRange() 

   Barry

> 
> Thanks,
> 
> Fande Kong,


From jed at jedbrown.org  Tue Jan  8 16:35:47 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 08 Jan 2019 15:35:47 -0700
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
Message-ID: <87o98qlqv0.fsf@jedbrown.org>

"Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:

>> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> Hi All,
>> 
>> I was wondering how to convert a regular (2D DMDA) grid coordinate (i,j) to a global index that is used for global matrices and vectors. Do we have any API on this? 
>
>     You can convert the global i,j to a local I,J by subtracting off the xs, ys, then convert to the local vector numbering with something like I+J*xm then add that value to the rstart returned from VecGetOwnershipRange() 

If the value may be in your ghosted/halo region, you can use

  VecGetLocalToGlobalMapping(vec, ltog);
  ISLocalToGlobalMappingApply(ltog, 1, &local, &global);

From bsmith at mcs.anl.gov  Tue Jan  8 16:40:00 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 8 Jan 2019 22:40:00 +0000
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <87o98qlqv0.fsf@jedbrown.org>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
	<87o98qlqv0.fsf@jedbrown.org>
Message-ID: <E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>



> On Jan 8, 2019, at 4:35 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
> 
>>> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi All,
>>> 
>>> I was wondering how to convert a regular (2D DMDA) grid coordinate (i,j) to a global index that is used for global matrices and vectors. Do we have any API on this? 
>> 
>>    You can convert the global i,j to a local I,J by subtracting off the xs, ys, then convert to the local vector numbering with something like I+J*xm then add that value to the rstart returned from VecGetOwnershipRange() 
> 
> If the value may be in your ghosted/halo region, you can use
> 
>  VecGetLocalToGlobalMapping(vec, ltog);
>  ISLocalToGlobalMappingApply(ltog, 1, &local, &global);

   Oh, yes I was ignoring the case where the questioning process didn't own the value. 

   In the general case where the i,j may not be within the ghost region of the process it is on you can use the ao obtained from DMDAGetAO() to map from the natural (application) ordering to the PETSc ordering. Obtain the natural ordering value with something like i + j*M. 


From fdkong.jd at gmail.com  Tue Jan  8 16:56:42 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 8 Jan 2019 15:56:42 -0700
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
	<87o98qlqv0.fsf@jedbrown.org>
	<E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
Message-ID: <CAN5Wd-L4ToQcRMNLvWve6fAhRHtpkT21dC-9S9k2Hp6+zsDRhQ@mail.gmail.com>

Thanks Barry and Jed,

I think I understood now.

One more question. We usually have a code pattern in some PETSc DMDA
examples as follows:

DMDAVecGetArrayRead(da,localU,&uarray);
DMDAVecGetArray(da,F,&f);

DMDAGetCorners(da,&xs,&ys,NULL,&xm,&ym,NULL);

for (j=ys; j<ys+ym; j++) {
    for (i=xs; i<xs+xm; i++) {\
      f[j][i] = uarray[j][i] - uarray[j-1][i];
}
}

Why we are using global i, j here? Should not we use i=i-xs, j=j-ys?
Aren't uarray and f  local arrays?

Thanks,

Fande Kong,




On Tue, Jan 8, 2019 at 3:40 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 8, 2019, at 4:35 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> > "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
> >
> >>> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how to convert a regular (2D DMDA) grid coordinate
> (i,j) to a global index that is used for global matrices and vectors. Do we
> have any API on this?
> >>
> >>    You can convert the global i,j to a local I,J by subtracting off the
> xs, ys, then convert to the local vector numbering with something like
> I+J*xm then add that value to the rstart returned from
> VecGetOwnershipRange()
> >
> > If the value may be in your ghosted/halo region, you can use
> >
> >  VecGetLocalToGlobalMapping(vec, ltog);
> >  ISLocalToGlobalMappingApply(ltog, 1, &local, &global);
>
>    Oh, yes I was ignoring the case where the questioning process didn't
> own the value.
>
>    In the general case where the i,j may not be within the ghost region of
> the process it is on you can use the ao obtained from DMDAGetAO() to map
> from the natural (application) ordering to the PETSc ordering. Obtain the
> natural ordering value with something like i + j*M.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/59eefb57/attachment-0001.html>

From fdkong.jd at gmail.com  Tue Jan  8 16:59:53 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 8 Jan 2019 15:59:53 -0700
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
	<87o98qlqv0.fsf@jedbrown.org>
	<E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
Message-ID: <CAN5Wd-JH0RBmXzp_aKLVWPgE8E=h3HU7B65JNDJ3emD6b7VEcw@mail.gmail.com>

On Tue, Jan 8, 2019 at 3:40 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 8, 2019, at 4:35 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> > "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
> >
> >>> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how to convert a regular (2D DMDA) grid coordinate
> (i,j) to a global index that is used for global matrices and vectors. Do we
> have any API on this?
> >>
> >>    You can convert the global i,j to a local I,J by subtracting off the
> xs, ys, then convert to the local vector numbering with something like
> I+J*xm then add that value to the rstart returned from
> VecGetOwnershipRange()
> >
> > If the value may be in your ghosted/halo region, you can use
> >
> >  VecGetLocalToGlobalMapping(vec, ltog);
> >  ISLocalToGlobalMappingApply(ltog, 1, &local, &global);
>
>    Oh, yes I was ignoring the case where the questioning process didn't
> own the value.
>
>    In the general case where the i,j may not be within the ghost region of
> the process it is on you can use the ao obtained from DMDAGetAO() to map
> from the natural (application) ordering to the PETSc ordering. Obtain the
> natural ordering value with something like i + j*M.
>

Is M the number of points (for all processors) along x direction?

Thanks,

Fande Kong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/46284b93/attachment.html>

From jed at jedbrown.org  Tue Jan  8 17:00:56 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 08 Jan 2019 16:00:56 -0700
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <CAN5Wd-L4ToQcRMNLvWve6fAhRHtpkT21dC-9S9k2Hp6+zsDRhQ@mail.gmail.com>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
	<87o98qlqv0.fsf@jedbrown.org>
	<E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
	<CAN5Wd-L4ToQcRMNLvWve6fAhRHtpkT21dC-9S9k2Hp6+zsDRhQ@mail.gmail.com>
Message-ID: <877efelpp3.fsf@jedbrown.org>

Fande Kong <fdkong.jd at gmail.com> writes:

> Thanks Barry and Jed,
>
> I think I understood now.
>
> One more question. We usually have a code pattern in some PETSc DMDA
> examples as follows:
>
> DMDAVecGetArrayRead(da,localU,&uarray);
> DMDAVecGetArray(da,F,&f);
>
> DMDAGetCorners(da,&xs,&ys,NULL,&xm,&ym,NULL);
>
> for (j=ys; j<ys+ym; j++) {
>     for (i=xs; i<xs+xm; i++) {\
>       f[j][i] = uarray[j][i] - uarray[j-1][i];
> }
> }
>
> Why we are using global i, j here? Should not we use i=i-xs, j=j-ys?
> Aren't uarray and f  local arrays?

DMDAVecGetArray (in contrast to VecGetArray) returns pointer-pointers
that have been set up to use global indices as u[j][i] instead of
something like uflat[(j-ys)*m+(i-xs)].

From bsmith at mcs.anl.gov  Tue Jan  8 17:03:00 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 8 Jan 2019 23:03:00 +0000
Subject: [petsc-users] In a 2D DMDA object, how to convert (i,
 j) to a global index?
In-Reply-To: <CAN5Wd-JH0RBmXzp_aKLVWPgE8E=h3HU7B65JNDJ3emD6b7VEcw@mail.gmail.com>
References: <CAN5Wd-L9NjCMJHuZzAffLf-4RJfRUt11TXBNbSxyKaJjHtCJ+A@mail.gmail.com>
	<A7533288-79E2-4D61-8047-C9C63E4C470B@anl.gov>
	<87o98qlqv0.fsf@jedbrown.org>
	<E09996E1-7F95-462A-A447-CC24D5B73046@mcs.anl.gov>
	<CAN5Wd-JH0RBmXzp_aKLVWPgE8E=h3HU7B65JNDJ3emD6b7VEcw@mail.gmail.com>
Message-ID: <DE3F85AC-CB59-4751-94CF-1043617F725C@mcs.anl.gov>


  Yes because the i,j are global indices into the entire array.


> On Jan 8, 2019, at 4:59 PM, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> 
> 
> On Tue, Jan 8, 2019 at 3:40 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
> > On Jan 8, 2019, at 4:35 PM, Jed Brown <jed at jedbrown.org> wrote:
> > 
> > "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
> > 
> >>> On Jan 8, 2019, at 4:11 PM, Fande Kong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> >>> 
> >>> Hi All,
> >>> 
> >>> I was wondering how to convert a regular (2D DMDA) grid coordinate (i,j) to a global index that is used for global matrices and vectors. Do we have any API on this? 
> >> 
> >>    You can convert the global i,j to a local I,J by subtracting off the xs, ys, then convert to the local vector numbering with something like I+J*xm then add that value to the rstart returned from VecGetOwnershipRange() 
> > 
> > If the value may be in your ghosted/halo region, you can use
> > 
> >  VecGetLocalToGlobalMapping(vec, ltog);
> >  ISLocalToGlobalMappingApply(ltog, 1, &local, &global);
> 
>    Oh, yes I was ignoring the case where the questioning process didn't own the value. 
> 
>    In the general case where the i,j may not be within the ghost region of the process it is on you can use the ao obtained from DMDAGetAO() to map from the natural (application) ordering to the PETSc ordering. Obtain the natural ordering value with something like i + j*M. 
> 
> Is M the number of points (for all processors) along x direction? 
> 
> Thanks,
> 
> Fande Kong
>  
>  


From olk548 at usask.ca  Tue Jan  8 23:40:25 2019
From: olk548 at usask.ca (Oleksandr Koshkarov)
Date: Wed, 9 Jan 2019 05:40:25 +0000
Subject: [petsc-users] Solve ODE in backward in time
In-Reply-To: <87h8elp3u9.fsf@jedbrown.org>
References: <YTOPR0101MB0713EF23D119977DA3B3C79BF0890@YTOPR0101MB0713.CANPRD01.PROD.OUTLOOK.COM>
	<87h8elp3u9.fsf@jedbrown.org>
Message-ID: <b0bba600-aa07-5f2b-7933-a32b6264df14@usask.ca>

Well, it is possible of course, but I thought petsc just has a simple 
switch. It looks like it does not

Thank you,

Oleksandr.

On 1/6/19 8:02 PM, Jed Brown wrote:
> Is it a problem to do the change of variables in your problem definition?
>
> "Koshkarov, Oleksandr via petsc-users" <petsc-users at mcs.anl.gov> writes:
>
>> Hello all,
>>
>> Can TSSolve be used to solve ODE backward in time? for example from t=10 upto t=5 ? Google search reveals to use TSAdjointSolve, but by reading manual, It looks it is used for sensitivity analysis which I do not need. And I am not sure how to use it to just integrate ODE back in time. I tried to setup final time and/or time step to be negative without success.
>>
>> Thank you and Happy New Year,
>>
>> Oleksandr.

From weston8 at llnl.gov  Wed Jan  9 16:53:27 2019
From: weston8 at llnl.gov (Weston, Brian Thomas)
Date: Wed, 9 Jan 2019 22:53:27 +0000
Subject: [petsc-users] Data types for local ID's and global ID's for large
 problems
Message-ID: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>

Does PETSc have different data types for local ID?s and global ID?s? For example, if PETSc is configured for 32-bit indices on very large problems with say 10 billion degrees of freedoms (exceeding the limit of 32-bit indices for a global ID) would it crash and require us to re-compile with 64-bit indices or can it gracefully handle this?

Sincerely,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190109/9e989bae/attachment.html>

From knepley at gmail.com  Wed Jan  9 16:55:26 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Jan 2019 17:55:26 -0500
Subject: [petsc-users] Data types for local ID's and global ID's for
 large problems
In-Reply-To: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>
References: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>
Message-ID: <CAMYG4GkhJU7gx-m2oPnoSBQJUa7D4jV7a03uNtma6aEbmAjZ9A@mail.gmail.com>

On Wed, Jan 9, 2019 at 5:53 PM Weston, Brian Thomas via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Does PETSc have different data types for local ID?s and global ID?s? For
> example, if PETSc is configured for 32-bit indices on very large problems
> with say 10 billion degrees of freedoms (exceeding the limit of 32-bit
> indices for a global ID) would it crash and require us to re-compile with
> 64-bit indices or can it gracefully handle this?
>

You would need to reconfigure for 64-bit indices. Keeping local indices
32-bit does not do much. I experimented with a local-index-only version
several years ago, but it breaks a lot of things, and does not really
deliver much benefit.

  Thanks,

    Matt


> Sincerely,
>
> Brian
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190109/bd65e09d/attachment.html>

From weston8 at llnl.gov  Wed Jan  9 18:07:52 2019
From: weston8 at llnl.gov (Weston, Brian Thomas)
Date: Thu, 10 Jan 2019 00:07:52 +0000
Subject: [petsc-users] Data types for local ID's and global ID's for
 large problems
In-Reply-To: <CAMYG4GkhJU7gx-m2oPnoSBQJUa7D4jV7a03uNtma6aEbmAjZ9A@mail.gmail.com>
References: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>
	<CAMYG4GkhJU7gx-m2oPnoSBQJUa7D4jV7a03uNtma6aEbmAjZ9A@mail.gmail.com>
Message-ID: <7F2ABE82-33B8-494C-B6DE-57B94B3C1EB0@llnl.gov>

Matt,

Thanks for the quick reply. Users of our ALE3D hydro code at LLNL run very large problems , which I believe can pass in large amounts of local integer data to solver packages like HYPRE and PETSc for global solves. All of our local integer data is stored with 32-bit indices and switching to 64-bit indices would increase the memory footprint and data motion. This is a problem when running on our BGQ machines like Sequoia machine and potentially for our GPU machines.

Is this not a performance issue for most PETSc users who are running very large problems on 64-bit indices? And is there any interest in the future for allowing local ID?s to be in 32-bit and global ID?s be in 64-bit?

Best,
Brian

From: Matthew Knepley <knepley at gmail.com>
Date: Wednesday, January 9, 2019 at 2:56 PM
To: "Weston, Brian Thomas" <weston8 at llnl.gov>
Cc: PETSc <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Data types for local ID's and global ID's for large problems

On Wed, Jan 9, 2019 at 5:53 PM Weston, Brian Thomas via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Does PETSc have different data types for local ID?s and global ID?s? For example, if PETSc is configured for 32-bit indices on very large problems with say 10 billion degrees of freedoms (exceeding the limit of 32-bit indices for a global ID) would it crash and require us to re-compile with 64-bit indices or can it gracefully handle this?

You would need to reconfigure for 64-bit indices. Keeping local indices 32-bit does not do much. I experimented with a local-index-only version several years ago, but it breaks a lot of things, and does not really deliver much benefit.

  Thanks,

    Matt

Sincerely,
Brian


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/b4eb6518/attachment.html>

From jed at jedbrown.org  Wed Jan  9 18:15:17 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 09 Jan 2019 17:15:17 -0700
Subject: [petsc-users] Data types for local ID's and global ID's for
 large problems
In-Reply-To: <7F2ABE82-33B8-494C-B6DE-57B94B3C1EB0@llnl.gov>
References: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>
	<CAMYG4GkhJU7gx-m2oPnoSBQJUa7D4jV7a03uNtma6aEbmAjZ9A@mail.gmail.com>
	<7F2ABE82-33B8-494C-B6DE-57B94B3C1EB0@llnl.gov>
Message-ID: <87imyx5pwq.fsf@jedbrown.org>

"Weston, Brian Thomas via petsc-users" <petsc-users at mcs.anl.gov> writes:

> Matt,
>
> Thanks for the quick reply. Users of our ALE3D hydro code at LLNL run very large problems , which I believe can pass in large amounts of local integer data to solver packages like HYPRE and PETSc for global solves. All of our local integer data is stored with 32-bit indices and switching to 64-bit indices would increase the memory footprint and data motion. This is a problem when running on our BGQ machines like Sequoia machine and potentially for our GPU machines.
>
> Is this not a performance issue for most PETSc users who are running very large problems on 64-bit indices? And is there any interest in the future for allowing local ID?s to be in 32-bit and global ID?s be in 64-bit?

The reason there is only one PetscInt is to reduce the number of
combinations to be tested (since C has weak typedefs).  The performance
impact observed in practice tends to be quite small.  PETSc might
consider internally changing the storage for common matrix types to use
a separate typedef so that PetscInt would be 64-bit but "PetscMatInt"
could be 32-bit, but I don't think anyone wants that to escape into the
public interface so it wouldn't have any impact on your code, just a
(slight) optimization --with-64-bit-indices.  (Note that some users need
64-bit on a single rank so it needs to be possible to use 64-bit
PetscMatInt.)

From tempohoper at gmail.com  Thu Jan 10 02:54:11 2019
From: tempohoper at gmail.com (Sal Am)
Date: Thu, 10 Jan 2019 08:54:11 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
Message-ID: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>

I am not sure what is exactly is wrong as the error changes slightly every
time I run it (without changing the parameters). I have attached the first
two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with
-malloc_dump, but not getting anything printed out, however, when run with
-log_view I see that Viewer is created 4 times, but destroyed 3 times. The
way I see it, I have destroyed it where I see I no longer have use for it
so not sure if I am wrong. Could this be the reason why it keeps crashing?
It crashes as soon as it reads the matrix, before entering the solving mode
(I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the
clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason
-ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
-ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/150ffe0f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: solveCmplxLinearSys.cpp
Type: text/x-c-code
Size: 5331 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/150ffe0f/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OneStripAntenna.e40896
Type: application/octet-stream
Size: 49301 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/150ffe0f/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OneStripAntenna.e40895
Type: application/octet-stream
Size: 49305 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/150ffe0f/attachment-0003.obj>

From dave.mayhem23 at gmail.com  Thu Jan 10 02:59:29 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Thu, 10 Jan 2019 08:59:29 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
Message-ID: <CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>

On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I am not sure what is exactly is wrong as the error changes slightly every
> time I run it (without changing the parameters).
>

This likely implies that you have a memory error in your code (a memory
leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.


> I have attached the first two run's errors and my code.
>
> Is there a memory leak somewhere? I have tried running it with
> -malloc_dump, but not getting anything printed out, however, when run with
> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
> way I see it, I have destroyed it where I see I no longer have use for it
> so not sure if I am wrong. Could this be the reason why it keeps crashing?
> It crashes as soon as it reads the matrix, before entering the solving mode
> (I have a print statement before solving starts that never prints).
>
> how I run it in the job script on 2 node with 32 processors using the
> clusters OpenMPI.
>
> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason
> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
> -ksp_monitor -malloc_log -ksp_view
>
> the matrix:
> 2 122 821 366 (non-zero elements)
> 25 947 279 x 25 947 279
>
> Thanks and all the best
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/28a36880/attachment.html>

From yann.jobic at univ-amu.fr  Thu Jan 10 05:46:59 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Thu, 10 Jan 2019 12:46:59 +0100
Subject: [petsc-users] DMPlex H27 elements
In-Reply-To: <3c4efe88-60e6-ff3b-9267-961cba5cbdb4@univ-amu.fr>
References: <fcd316d2-b94f-9659-de77-c3c3ad8dc813@univ-amu.fr>
	<CAMYG4GngZgMPtcpjPfpxSQ66UEaFrXgSigi-bXa8OHF9Z_JkVg@mail.gmail.com>
	<498c2ec0-5859-f1a1-79ec-8be415d70596@univ-amu.fr>
	<CAMYG4G==s1+9o3Eto08g-hF7JgwVLnxovMzs_t_Tpp44b4k9OQ@mail.gmail.com>
	<3c4efe88-60e6-ff3b-9267-961cba5cbdb4@univ-amu.fr>
Message-ID: <09fce6bf-6d41-2d20-9eae-7d52e0c5ec85@univ-amu.fr>


On 08/01/2019 19:45, Yann Jobic via petsc-users wrote:
>
>
> On 05/01/2019 13:23, Matthew Knepley wrote:
>> On Sat, Jan 5, 2019 at 4:04 AM Yann Jobic <yann.jobic at univ-amu.fr 
>> <mailto:yann.jobic at univ-amu.fr>> wrote:
>>
>>
>>     On 05/01/2019 02:36, Matthew Knepley wrote:
>>>     On Fri, Jan 4, 2019 at 10:04 AM Yann Jobic via petsc-users
>>>     <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>
>>>         Dear Petsc Users,
>>>
>>>         I'm using DMPlexCreateFromCellList to create my DM. I would
>>>         like to have
>>>         an order 2 geometry.
>>>
>>>         It's working fine in 2D for elements of type Q9.
>>>
>>>
>>>     I do not see how that is possible. The topology does not work
>>>     that way, and there
>>>     is no way a 9 vertex quad was interpolated, so something else is
>>>     happening.
>>>
>>>     Moreover, anything you input still has affine geometry. Toby has
>>>     been working on
>>>     non-affine geometry, and all the code is there, but right now
>>>     you have to build and
>>>     populate the coordinate space by hand. What this means is that
>>>     you create a
>>>     quadratic space for the coordinate DM (rather than the default
>>>     linear), get the
>>>     global coordinate vector out, and stick in the Q9 coordinates.
>>>     The topology is still
>>>     the same no matter what coordinates you read in.
>>
>>     I get the idea. So at the beginning i start with the Q4/H8
>>     dmplex, then i fill from the global coordinate vector the missing
>>     coordinates, and attach this new coordinate vector to the DM.
>>
>> Yes.
>
> So far, i can change the PetscSpace order, add this space to a 
> PetscFE, and finally attach this PetscFE to a PetscDS (discetization) 
> to my DM.
>
> But if i'm doing that, the middle nodes will already be added (i 
> couldn't find this function so far, could you point me the name ?). I 
> can still move those middle vertex to my geometry.
>
> But i couldn't find a way to define the quadratic space for the 
> coordinate DM before all of that, at the DM (or DMPlex) level.
>
> What way can you advise me to use in order to do that ?
>
I think i got it, if i'm not wrong. I mean, the first step. It was in 
the documentation...

I defined the PetscSection according to the element i use. The size of 
the matrix coming from the DM is correct.

I'm working on the next step : "populate the space coordinates".

Yann

> Thank you,
>
> Yann
>
>>     Then the interpolation will be correct ?
>>
>> This is the tricky part. What you likely have to do is:
>>
>> ? 1) Build the topology
>> ? 2) Run over the cells
>> ? 3) For each cell, put in the coordinates using 
>> DMPlexVecSetClosure(). This is redundant, but the only way
>> ? ? ? you can connect with the Q9 format.
>>
>>     Can i use DMproject and all the petsc functions ?
>>
>> This is the idea. The amount of testing is so far almost nothing.
>>
>>     In order to output the results, i've got to write my own vtk
>>     interface no ? (by splitting the Q9/H27 in 4xQ4 or 8xH8).
>>
>> That is a good question. Right now, it will just output Q4. However, 
>> compared to everything else, writing Q9 should
>> be comparatively easy.
>>
>>     Or is it working with the hdf5 one ?
>>
>> I would probably write code to to Xdmf through HDF5 rather than 
>> straight VTK, but we will end up having both.
>>
>>     Does the p4est interface work with this setup ?
>>
>> It will. I do not know if it currently does. We have to check.
>>
>> ? Thanks,
>>
>> ? ? Matt
>>
>>>
>>>     We will get around to making an interface for this soon.
>>
>>     Great !
>>
>>     Thanks,
>>
>>     Yann
>>
>>>
>>>     ? Thanks,
>>>
>>>     ? ? ?Matt
>>>
>>>         I checked that it's working correctly by using
>>>         DMPlexComputeCellGeometryFEM, and compute the value of the
>>>         determinant
>>>         (CheckMeshGeometry of dm/impls/plex/examples/tutorials/ex2.c)
>>>
>>>         I can import H8 elements, it's working fine.
>>>
>>>         But for H27 element, it's not working. I really don't know
>>>         how to order
>>>         my vertex in the cells array. So far, the determinant is zero...
>>>
>>>         I don't know where to look in order to find this
>>>         information. I tried
>>>         the function DMPlexGetRawFaces_Internal of the file
>>>
>>>         https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/plexinterpolate.c.html#DMPlexGetRawFaces_Internal
>>>
>>>         but it didn't help me.
>>>
>>>         Could you please point me where to look ?
>>>
>>>         Thanks, and happy new year !
>>>
>>>         Yann
>>>
>>>
>>>
>>>     -- 
>>>     What most experimenters take for granted before they begin their
>>>     experiments is infinitely more interesting than any results to
>>>     which their experiments lead.
>>>     -- Norbert Wiener
>>>
>>>     https://www.cse.buffalo.edu/~knepley/
>>>     <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their 
>> experiments is infinitely more interesting than any results to which 
>> their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/ 
>> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/4d49f526/attachment.html>

From overholt at capesim.com  Thu Jan 10 08:43:21 2019
From: overholt at capesim.com (Matthew Overholt)
Date: Thu, 10 Jan 2019 09:43:21 -0500
Subject: [petsc-users] METIS double precision
Message-ID: <CAMSLWkvWDwVruJQu_c8jDes7sZeBpiTiGwZ3AYzN_Mo_BftUoA@mail.gmail.com>

Hello,

How does one configure the PETSc installation to download METIS and have it
use REALTYPEWIDTH 64 (as defined in metis.h)?  I am using:

--with-64-bit-indices --download-metis=yes

to get IDXTYPEWIDTH 64.  If I were installing METIS independently I would
set the following near the top of metis.h:  #define
METIS_USE_DOUBLEPRECISION

The reason for this is I want to call METIS routines outside of PETSc and
prefer double precision.

Thanks,
Matt Overholt
CapeSym, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/61e863fa/attachment-0001.html>

From balay at mcs.anl.gov  Thu Jan 10 09:35:27 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 10 Jan 2019 15:35:27 +0000
Subject: [petsc-users] METIS double precision
In-Reply-To: <CAMSLWkvWDwVruJQu_c8jDes7sZeBpiTiGwZ3AYzN_Mo_BftUoA@mail.gmail.com>
References: <CAMSLWkvWDwVruJQu_c8jDes7sZeBpiTiGwZ3AYzN_Mo_BftUoA@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901100932420.12710@sb>

Well - this was the previous default - and was removed as it was deemed unnecessary.

https://bitbucket.org/petsc/petsc/commits/2d4f01c230fe350f0ab5a28d1f5ef05ceab7ea3d

The attached patch provides --download-metis-use-doubleprecision option

Satish

On Thu, 10 Jan 2019, Matthew Overholt via petsc-users wrote:

> Hello,
> 
> How does one configure the PETSc installation to download METIS and have it
> use REALTYPEWIDTH 64 (as defined in metis.h)?  I am using:
> 
> --with-64-bit-indices --download-metis=yes
> 
> to get IDXTYPEWIDTH 64.  If I were installing METIS independently I would
> set the following near the top of metis.h:  #define
> METIS_USE_DOUBLEPRECISION
> 
> The reason for this is I want to call METIS routines outside of PETSc and
> prefer double precision.
> 
> Thanks,
> Matt Overholt
> CapeSym, Inc.
> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: metis-doubleprecision.patch
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/c2ccd1c4/attachment.ksh>

From balay at mcs.anl.gov  Thu Jan 10 09:45:15 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 10 Jan 2019 15:45:15 +0000
Subject: [petsc-users] METIS double precision
In-Reply-To: <alpine.LFD.2.21.1901100932420.12710@sb>
References: <CAMSLWkvWDwVruJQu_c8jDes7sZeBpiTiGwZ3AYzN_Mo_BftUoA@mail.gmail.com>
	<alpine.LFD.2.21.1901100932420.12710@sb>
Message-ID: <alpine.LFD.2.21.1901100945010.12710@sb>

https://bitbucket.org/petsc/petsc/pull-requests/1316/metis-provide-download-metis-use/

Satish

On Thu, 10 Jan 2019, Balay, Satish via petsc-users wrote:

> Well - this was the previous default - and was removed as it was deemed unnecessary.
> 
> https://bitbucket.org/petsc/petsc/commits/2d4f01c230fe350f0ab5a28d1f5ef05ceab7ea3d
> 
> The attached patch provides --download-metis-use-doubleprecision option
> 
> Satish
> 
> On Thu, 10 Jan 2019, Matthew Overholt via petsc-users wrote:
> 
> > Hello,
> > 
> > How does one configure the PETSc installation to download METIS and have it
> > use REALTYPEWIDTH 64 (as defined in metis.h)?  I am using:
> > 
> > --with-64-bit-indices --download-metis=yes
> > 
> > to get IDXTYPEWIDTH 64.  If I were installing METIS independently I would
> > set the following near the top of metis.h:  #define
> > METIS_USE_DOUBLEPRECISION
> > 
> > The reason for this is I want to call METIS routines outside of PETSc and
> > prefer double precision.
> > 
> > Thanks,
> > Matt Overholt
> > CapeSym, Inc.
> > 
> 


From overholt at capesim.com  Thu Jan 10 11:27:10 2019
From: overholt at capesim.com (Matthew Overholt)
Date: Thu, 10 Jan 2019 12:27:10 -0500
Subject: [petsc-users] METIS double precision
In-Reply-To: <alpine.LFD.2.21.1901100932420.12710@sb>
References: <CAMSLWkvWDwVruJQu_c8jDes7sZeBpiTiGwZ3AYzN_Mo_BftUoA@mail.gmail.com>
	<alpine.LFD.2.21.1901100932420.12710@sb>
Message-ID: <CAMSLWkvFnd1+djLcXQfEbLuLi8LLULROcBfA17ak9QwAZAnUaQ@mail.gmail.com>

Thank you, Satish!  That edit was easy to make and the configured and built
library works perfectly!

Just a note for new users: the patch to metis.py does not replace the
original --download-metis=<bool> argument, but supplements it (i.e. use
both).  I made this mistake on my first try and METIS was not downloaded.

Matt Overholt



On Thu, Jan 10, 2019 at 10:35 AM Balay, Satish <balay at mcs.anl.gov> wrote:

> Well - this was the previous default - and was removed as it was deemed
> unnecessary.
>
>
> https://bitbucket.org/petsc/petsc/commits/2d4f01c230fe350f0ab5a28d1f5ef05ceab7ea3d
>
> The attached patch provides --download-metis-use-doubleprecision option
>
> Satish
>
> On Thu, 10 Jan 2019, Matthew Overholt via petsc-users wrote:
>
> > Hello,
> >
> > How does one configure the PETSc installation to download METIS and have
> it
> > use REALTYPEWIDTH 64 (as defined in metis.h)?  I am using:
> >
> > --with-64-bit-indices --download-metis=yes
> >
> > to get IDXTYPEWIDTH 64.  If I were installing METIS independently I would
> > set the following near the top of metis.h:  #define
> > METIS_USE_DOUBLEPRECISION
> >
> > The reason for this is I want to call METIS routines outside of PETSc and
> > prefer double precision.
> >
> > Thanks,
> > Matt Overholt
> > CapeSym, Inc.
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/87d9750c/attachment.html>

From knepley at gmail.com  Thu Jan 10 11:52:46 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Jan 2019 12:52:46 -0500
Subject: [petsc-users] Data types for local ID's and global ID's for
 large problems
In-Reply-To: <87imyx5pwq.fsf@jedbrown.org>
References: <4691DC0D-D27D-4213-BE3F-CC3505B32D6B@llnl.gov>
	<CAMYG4GkhJU7gx-m2oPnoSBQJUa7D4jV7a03uNtma6aEbmAjZ9A@mail.gmail.com>
	<7F2ABE82-33B8-494C-B6DE-57B94B3C1EB0@llnl.gov>
	<87imyx5pwq.fsf@jedbrown.org>
Message-ID: <CAMYG4GkYmR5HCe2ttCyDCkazXzAhpRgauYPu_3Jriu66BXoZFQ@mail.gmail.com>

On Wed, Jan 9, 2019 at 7:15 PM Jed Brown <jed at jedbrown.org> wrote:

> "Weston, Brian Thomas via petsc-users" <petsc-users at mcs.anl.gov> writes:
>
> > Matt,
> >
> > Thanks for the quick reply. Users of our ALE3D hydro code at LLNL run
> very large problems , which I believe can pass in large amounts of local
> integer data to solver packages like HYPRE and PETSc for global solves. All
> of our local integer data is stored with 32-bit indices and switching to
> 64-bit indices would increase the memory footprint and data motion. This is
> a problem when running on our BGQ machines like Sequoia machine and
> potentially for our GPU machines.
> >
> > Is this not a performance issue for most PETSc users who are running
> very large problems on 64-bit indices? And is there any interest in the
> future for allowing local ID?s to be in 32-bit and global ID?s be in 64-bit?
>
> The reason there is only one PetscInt is to reduce the number of
> combinations to be tested (since C has weak typedefs).  The performance
> impact observed in practice tends to be quite small.  PETSc might
> consider internally changing the storage for common matrix types to use
> a separate typedef so that PetscInt would be 64-bit but "PetscMatInt"
> could be 32-bit, but I don't think anyone wants that to escape into the
> public interface so it wouldn't have any impact on your code, just a
> (slight) optimization --with-64-bit-indices.  (Note that some users need
> 64-bit on a single rank so it needs to be possible to use 64-bit
> PetscMatInt.)
>

As Jed says, all our tests show a negligible difference. However, if you
had tests on BGQ that showed a big impact, we would love to see the
numbers. That could justify work on this.

  Thanks,

     Matt
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/8d8186cd/attachment.html>

From yaqiwang at gmail.com  Thu Jan 10 14:29:14 2019
From: yaqiwang at gmail.com (Yaqi Wang)
Date: Thu, 10 Jan 2019 13:29:14 -0700
Subject: [petsc-users] PETSc configure error
Message-ID: <CACL+gcP1OyBwwYg5EXCmPnt=vWK+J0rLBAkzbMdeEDGXPuT2nQ@mail.gmail.com>

Hello,

I am having difficulty on configuring PETSc. Running configure crashed in
the middle. At the end of configure.log, I have
---------------

*******************************************************************************

        CONFIGURATION CRASH  (Please send configure.log to
petsc-maint at mcs.anl.gov)

*******************************************************************************

'NoneType' object has no attribute 'group'  File "./config/configure.py",
line 394, in petsc_configure

    framework.configure(out = sys.stdout)

  File
"/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
line 1092, in configure

    self.processChildren()

  File
"/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
line 1081, in processChildren

    self.serialEvaluation(self.childGraph)

  File
"/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
line 1062, in serialEvaluation

    child.configure()

  File
"/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
line 119, in configure

    self.checkBfortVersion(1,1,25)

  File
"/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
line 59, in checkBfortVersion

    major    = int(ver.group(1))

================================================================================

Finishing configure run at Thu, 10 Jan 2019 13:04:21 -0700

================================================================================

----------

Does anyone have the similar error before? How should I fix it?


Your help is appreciated.


Best,

Yaqi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/93fe095d/attachment.html>

From balay at mcs.anl.gov  Thu Jan 10 14:54:45 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 10 Jan 2019 20:54:45 +0000
Subject: [petsc-users] PETSc configure error
In-Reply-To: <CACL+gcP1OyBwwYg5EXCmPnt=vWK+J0rLBAkzbMdeEDGXPuT2nQ@mail.gmail.com>
References: <CACL+gcP1OyBwwYg5EXCmPnt=vWK+J0rLBAkzbMdeEDGXPuT2nQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1901101454160.12710@sb>

Can you send configure.log from this failure?

Do you have any old version of bfort in your PATH?

Satish

On Thu, 10 Jan 2019, Yaqi Wang via petsc-users wrote:

> Hello,
> 
> I am having difficulty on configuring PETSc. Running configure crashed in
> the middle. At the end of configure.log, I have
> ---------------
> 
> *******************************************************************************
> 
>         CONFIGURATION CRASH  (Please send configure.log to
> petsc-maint at mcs.anl.gov)
> 
> *******************************************************************************
> 
> 'NoneType' object has no attribute 'group'  File "./config/configure.py",
> line 394, in petsc_configure
> 
>     framework.configure(out = sys.stdout)
> 
>   File
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> line 1092, in configure
> 
>     self.processChildren()
> 
>   File
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> line 1081, in processChildren
> 
>     self.serialEvaluation(self.childGraph)
> 
>   File
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> line 1062, in serialEvaluation
> 
>     child.configure()
> 
>   File
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
> line 119, in configure
> 
>     self.checkBfortVersion(1,1,25)
> 
>   File
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
> line 59, in checkBfortVersion
> 
>     major    = int(ver.group(1))
> 
> ================================================================================
> 
> Finishing configure run at Thu, 10 Jan 2019 13:04:21 -0700
> 
> ================================================================================
> 
> ----------
> 
> Does anyone have the similar error before? How should I fix it?
> 
> 
> Your help is appreciated.
> 
> 
> Best,
> 
> Yaqi
> 


From yaqiwang at gmail.com  Thu Jan 10 15:14:53 2019
From: yaqiwang at gmail.com (Yaqi Wang)
Date: Thu, 10 Jan 2019 14:14:53 -0700
Subject: [petsc-users] PETSc configure error
In-Reply-To: <alpine.LFD.2.21.1901101454160.12710@sb>
References: <CACL+gcP1OyBwwYg5EXCmPnt=vWK+J0rLBAkzbMdeEDGXPuT2nQ@mail.gmail.com>
	<alpine.LFD.2.21.1901101454160.12710@sb>
Message-ID: <CACL+gcNrbhRXqZe11cm90zbjjdYF2gU6wcBhb9TUcZ5+8WR6Qw@mail.gmail.com>

Satish,

Yes, indeed. I figured my bfort is somehow corrupted. 'otool -L' shows that
it is pointing to an non-existing MPI library. Using install_name_tool on
bfort fixed the problem.

Best,
Yaqi

On Thu, Jan 10, 2019 at 1:54 PM Balay, Satish <balay at mcs.anl.gov> wrote:

> Can you send configure.log from this failure?
>
> Do you have any old version of bfort in your PATH?
>
> Satish
>
> On Thu, 10 Jan 2019, Yaqi Wang via petsc-users wrote:
>
> > Hello,
> >
> > I am having difficulty on configuring PETSc. Running configure crashed in
> > the middle. At the end of configure.log, I have
> > ---------------
> >
> >
> *******************************************************************************
> >
> >         CONFIGURATION CRASH  (Please send configure.log to
> > petsc-maint at mcs.anl.gov)
> >
> >
> *******************************************************************************
> >
> > 'NoneType' object has no attribute 'group'  File "./config/configure.py",
> > line 394, in petsc_configure
> >
> >     framework.configure(out = sys.stdout)
> >
> >   File
> >
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> > line 1092, in configure
> >
> >     self.processChildren()
> >
> >   File
> >
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> > line 1081, in processChildren
> >
> >     self.serialEvaluation(self.childGraph)
> >
> >   File
> >
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/framework.py",
> > line 1062, in serialEvaluation
> >
> >     child.configure()
> >
> >   File
> >
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
> > line 119, in configure
> >
> >     self.checkBfortVersion(1,1,25)
> >
> >   File
> >
> "/Users/wangy2/projects/moose/petsc/config/BuildSystem/config/packages/sowing.py",
> > line 59, in checkBfortVersion
> >
> >     major    = int(ver.group(1))
> >
> >
> ================================================================================
> >
> > Finishing configure run at Thu, 10 Jan 2019 13:04:21 -0700
> >
> >
> ================================================================================
> >
> > ----------
> >
> > Does anyone have the similar error before? How should I fix it?
> >
> >
> > Your help is appreciated.
> >
> >
> > Best,
> >
> > Yaqi
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/8dcc11b5/attachment.html>

From fdkong.jd at gmail.com  Thu Jan 10 16:09:27 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Thu, 10 Jan 2019 15:09:27 -0700
Subject: [petsc-users] Any reason for API change: DMGetWorkArray()
Message-ID: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>

Hi All,

The second parameter is changed from PetscDataType to MPI_Datatype starting
from PETSc-3.9.x

Thanks,

Fande Kong,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/db18d846/attachment.html>

From knepley at gmail.com  Thu Jan 10 16:21:59 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Jan 2019 17:21:59 -0500
Subject: [petsc-users] Any reason for API change: DMGetWorkArray()
In-Reply-To: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>
References: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>
Message-ID: <CAMYG4Gkt9=dEWYPsMAz6QHeo_GPR9AEnZBtUHDO02HoK2ZVRhQ@mail.gmail.com>

We are trying to eliminate PetscDataType.

  Matt

On Thu, Jan 10, 2019 at 5:10 PM Fande Kong via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi All,
>
> The second parameter is changed from PetscDataType to MPI_Datatype
> starting from PETSc-3.9.x
>
> Thanks,
>
> Fande Kong,
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/27d4aa61/attachment-0001.html>

From fdkong.jd at gmail.com  Thu Jan 10 16:31:47 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Thu, 10 Jan 2019 15:31:47 -0700
Subject: [petsc-users] Any reason for API change: DMGetWorkArray()
In-Reply-To: <CAMYG4Gkt9=dEWYPsMAz6QHeo_GPR9AEnZBtUHDO02HoK2ZVRhQ@mail.gmail.com>
References: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>
	<CAMYG4Gkt9=dEWYPsMAz6QHeo_GPR9AEnZBtUHDO02HoK2ZVRhQ@mail.gmail.com>
Message-ID: <CAN5Wd-KxR0gUfJ71C3RANpGgmN+_6Fka08gDRzPnOyCRhwVFVw@mail.gmail.com>

Thanks, Matt,

And then what is the reason to remove PetscDataType? I am out of curiosity.

DMGetWorkArray is a little misleading. This may/might make people think the
routine is related to MPI, but it does not have anything to do with MPI.

Thanks,

Fande,

On Thu, Jan 10, 2019 at 3:22 PM Matthew Knepley <knepley at gmail.com> wrote:

> We are trying to eliminate PetscDataType.
>
>   Matt
>
> On Thu, Jan 10, 2019 at 5:10 PM Fande Kong via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi All,
>>
>> The second parameter is changed from PetscDataType to MPI_Datatype
>> starting from PETSc-3.9.x
>>
>> Thanks,
>>
>> Fande Kong,
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/3410b6eb/attachment.html>

From knepley at gmail.com  Thu Jan 10 16:36:21 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Jan 2019 17:36:21 -0500
Subject: [petsc-users] Any reason for API change: DMGetWorkArray()
In-Reply-To: <CAN5Wd-KxR0gUfJ71C3RANpGgmN+_6Fka08gDRzPnOyCRhwVFVw@mail.gmail.com>
References: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>
	<CAMYG4Gkt9=dEWYPsMAz6QHeo_GPR9AEnZBtUHDO02HoK2ZVRhQ@mail.gmail.com>
	<CAN5Wd-KxR0gUfJ71C3RANpGgmN+_6Fka08gDRzPnOyCRhwVFVw@mail.gmail.com>
Message-ID: <CAMYG4GkVL_jnrCDu784N-vPiX8M-U=k4kNBSEvLw3LsVu2tjLA@mail.gmail.com>

On Thu, Jan 10, 2019 at 5:31 PM Fande Kong <fdkong.jd at gmail.com> wrote:

> Thanks, Matt,
>
> And then what is the reason to remove PetscDataType? I am out of curiosity.
>

Occam's Razor: "one should not increase, beyond what is necessary, the
number of entities required to explain anything"

   Matt


> DMGetWorkArray is a little misleading. This may/might make people think
> the routine is related to MPI, but it does not have anything to do with MPI.
>
> Thanks,
>
> Fande,
>
> On Thu, Jan 10, 2019 at 3:22 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> We are trying to eliminate PetscDataType.
>>
>>   Matt
>>
>> On Thu, Jan 10, 2019 at 5:10 PM Fande Kong via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi All,
>>>
>>> The second parameter is changed from PetscDataType to MPI_Datatype
>>> starting from PETSc-3.9.x
>>>
>>> Thanks,
>>>
>>> Fande Kong,
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/9c4a526f/attachment.html>

From fdkong.jd at gmail.com  Thu Jan 10 17:07:22 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Thu, 10 Jan 2019 16:07:22 -0700
Subject: [petsc-users] Any reason for API change: DMGetWorkArray()
In-Reply-To: <CAMYG4GkVL_jnrCDu784N-vPiX8M-U=k4kNBSEvLw3LsVu2tjLA@mail.gmail.com>
References: <CAN5Wd-K+W_8kTf6c1-6Yp2oLL+SS54KYxE97922yu0obbQ4a1A@mail.gmail.com>
	<CAMYG4Gkt9=dEWYPsMAz6QHeo_GPR9AEnZBtUHDO02HoK2ZVRhQ@mail.gmail.com>
	<CAN5Wd-KxR0gUfJ71C3RANpGgmN+_6Fka08gDRzPnOyCRhwVFVw@mail.gmail.com>
	<CAMYG4GkVL_jnrCDu784N-vPiX8M-U=k4kNBSEvLw3LsVu2tjLA@mail.gmail.com>
Message-ID: <CAN5Wd-JYL9hh17kzQU2KXXySpZMJceFRk5mJM7ZQRLBmrZwbBQ@mail.gmail.com>

OK...,

Thanks for the words.

Fande,

On Thu, Jan 10, 2019 at 3:36 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 10, 2019 at 5:31 PM Fande Kong <fdkong.jd at gmail.com> wrote:
>
>> Thanks, Matt,
>>
>> And then what is the reason to remove PetscDataType? I am out of
>> curiosity.
>>
>
> Occam's Razor: "one should not increase, beyond what is necessary, the
> number of entities required to explain anything"
>
>    Matt
>
>
>> DMGetWorkArray is a little misleading. This may/might make people think
>> the routine is related to MPI, but it does not have anything to do with MPI.
>>
>> Thanks,
>>
>> Fande,
>>
>> On Thu, Jan 10, 2019 at 3:22 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> We are trying to eliminate PetscDataType.
>>>
>>>   Matt
>>>
>>> On Thu, Jan 10, 2019 at 5:10 PM Fande Kong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi All,
>>>>
>>>> The second parameter is changed from PetscDataType to MPI_Datatype
>>>> starting from PETSc-3.9.x
>>>>
>>>> Thanks,
>>>>
>>>> Fande Kong,
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190110/2c65d072/attachment.html>

From leejearl at mail.nwpu.edu.cn  Fri Jan 11 02:35:36 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Fri, 11 Jan 2019 16:35:36 +0800
Subject: [petsc-users] How Can I scatter a global vector to a local vector
 without using DMGlobaltoLocalBegin/DMGlobaltoLocalEnd?
Message-ID: <e88b97f0457316cbbefb049c6344b9dbc6580f80.camel@mail.nwpu.edu.cn>

Hi, all Petscer:
  I am very sorry for disturbing you for my questions. I have a
problems as follow.
  1. A DMPlex object dm is created.
  2. A global vector and a local vector are created using the following
routines DMCreateGlobalVector and DMCreateLocalVector.
  3. I have initialed global Vector. Because the local vector have some
overlap cells, I want to scatter the data from global to local.
  4. If I don't use the routines
DMGlobaltoLocalBegin/DMGlobaltoLocalEnd, how can I scatter it? I have
scatter the vectors using VecScatterBegin and VecScatterEnd, and the IS
is created using the localId and the glolbalId (Obtained by
DMPlexGetCellNumbering). I got a wrong result.

  Can anyone give me some advices, and does it have some to do with the
natural ids?
  Any helps are approciated. Thanks. 





leejearl


From tempohoper at gmail.com  Fri Jan 11 02:41:05 2019
From: tempohoper at gmail.com (Sal Am)
Date: Fri, 11 Jan 2019 08:41:05 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
Message-ID: <CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>

Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again
with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting
valgrind.log.%p files (all 8 corresponding to each processor) are all
empty. And it took a whooping ~15hours, for what used to take ~10-20min.
Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com> wrote:

>
>
> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I am not sure what is exactly is wrong as the error changes slightly
>> every time I run it (without changing the parameters).
>>
>
> This likely implies that you have a memory error in your code (a memory
> leak would not cause this behaviour).
> I strongly suggest you make sure your code is free of memory errors.
> You can do this using valgrind. See here
>
> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
> for an explanation of how to use valgrind.
>
>
>> I have attached the first two run's errors and my code.
>>
>> Is there a memory leak somewhere? I have tried running it with
>> -malloc_dump, but not getting anything printed out, however, when run with
>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>> way I see it, I have destroyed it where I see I no longer have use for it
>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>> It crashes as soon as it reads the matrix, before entering the solving mode
>> (I have a print statement before solving starts that never prints).
>>
>> how I run it in the job script on 2 node with 32 processors using the
>> clusters OpenMPI.
>>
>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason
>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>> -ksp_monitor -malloc_log -ksp_view
>>
>> the matrix:
>> 2 122 821 366 (non-zero elements)
>> 25 947 279 x 25 947 279
>>
>> Thanks and all the best
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/3777c6da/attachment-0001.html>
-------------- next part --------------
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------



      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


./solveCSys on a linux-cumulus-debug named r02g03 with 8 processors, by vef002 Fri Jan 11 01:58:39 2019
Using Petsc Release Version 3.10.2, unknown 

                         Max       Max/Min     Avg       Total 
Time (sec):           5.385e+04     1.000   5.385e+04
Objects:              2.880e+02     1.003   2.871e+02
Flop:                 1.192e+12     1.425   9.458e+11  7.567e+12
Flop/sec:             2.214e+07     1.425   1.756e+07  1.405e+08
MPI Messages:         3.749e+05     1.927   3.001e+05  2.401e+06
MPI Message Lengths:  6.445e+09     3.656   1.605e+04  3.853e+10
MPI Reductions:       2.571e+05     1.000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flop
                            and VecAXPY() for complex vectors of length N --> 8N flop

Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total    Count   %Total     Avg         %Total    Count   %Total 
 0:      Main Stage: 5.3850e+04 100.0%  7.5667e+12 100.0%  2.401e+06 100.0%  1.605e+04      100.0%  2.571e+05 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
                  Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   AvgLen: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flop in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          2 1.0 1.0046e-02 1.0 0.00e+00 0.0 4.1e+01 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BuildTwoSidedF        23 1.0 8.2332e+01237.1 0.00e+00 0.0 2.0e+02 4.5e+06 0.0e+00  0  0  0  2  0   0  0  0  2  0     0
VecView                1 1.0 1.7768e-01 1.0 0.00e+00 0.0 7.0e+00 2.8e+05 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecDot              4736 1.0 1.5488e+01 1.0 6.61e+08 1.0 0.0e+00 0.0e+00 9.5e+03  0  0  0  0  4   0  0  0  0  4   341
VecDotNorm2         2368 1.0 1.3977e+01 1.0 6.61e+08 1.0 0.0e+00 0.0e+00 4.7e+03  0  0  0  0  2   0  0  0  0  2   378
VecMDot               40 1.0 6.6729e+0011.1 1.54e+07 1.0 0.0e+00 0.0e+00 8.0e+01  0  0  0  0  0   0  0  0  0  0    18
VecNorm             2413 1.0 8.1465e+00 1.0 3.34e+08 1.0 0.0e+00 0.0e+00 4.8e+03  0  0  0  0  2   0  0  0  0  2   328
VecScale              44 1.0 4.3048e-02 1.0 1.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   287
VecCopy             9480 1.0 1.7119e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             42668 1.0 2.5871e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                4 1.0 9.8097e-03 1.0 2.81e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   229
VecAYPX            75792 1.0 5.5978e+01 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   475
VecAXPBYCZ         42632 1.0 1.5645e+02 1.0 7.97e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0   407
VecWAXPY            4736 1.0 1.3468e+01 1.0 6.61e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   393
VecMAXPY              44 1.0 3.4021e-01 1.0 1.82e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   429
VecAssemblyBegin      10 1.0 2.0956e-01 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd        10 1.0 1.2929e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      22 1.0 3.1633e-02 1.0 7.72e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   195
VecLoad                1 1.0 3.6724e-01 1.0 0.00e+00 0.0 7.0e+00 2.8e+05 1.1e+01  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin    80580 1.0 2.8643e+01 2.3 0.00e+00 0.0 2.4e+06 1.6e+04 0.0e+00  0  0100 97  0   0  0100 97  0     0
VecScatterEnd      80580 1.0 6.2659e+03270.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   3  0  0  0  0     0
VecSetRandom           2 1.0 1.7043e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          44 1.0 4.3032e-01 3.1 4.63e+06 1.0 0.0e+00 0.0e+00 8.8e+01  0  0  0  0  0   0  0  0  0  0    86
MatMult            61620 1.0 2.9162e+04 1.3 6.48e+11 1.4 2.1e+06 1.8e+04 0.0e+00 46 56 87 96  0  46 56 87 96  0   145
MatMultAdd          9474 1.0 1.5750e+02 1.2 3.28e+09 1.3 1.6e+05 4.4e+02 0.0e+00  0  0  7  0  0   0  0  7  0  0   152
MatMultTranspose    9474 1.0 1.1072e+02 1.1 3.28e+09 1.3 1.6e+05 4.4e+02 9.5e+03  0  0  7  0  4   0  0  7  0  4   216
MatSolve            4737 0.0 2.3544e+00 0.0 2.62e+07 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    11
MatSOR             56866 1.0 2.3703e+04 1.4 5.21e+11 1.6 0.0e+00 0.0e+00 0.0e+00 35 42  0  0  0  35 42  0  0  0   133
MatLUFactorSym         1 1.0 2.6810e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 4.8278e-02 8.7 4.67e+04 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     1
MatConvert             2 1.0 1.2898e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               6 1.0 7.0816e-01 1.2 2.02e+07 1.3 7.0e+01 1.6e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0   188
MatResidual         9474 1.0 4.1742e+03 1.0 9.29e+10 1.3 3.3e+05 1.6e+04 9.5e+03  8  8 14 14  4   8  8 14 14  4   145
MatAssemblyBegin      47 1.0 1.4260e+02 1.4 0.00e+00 0.0 2.0e+02 4.5e+06 2.8e+01  0  0  0  2  0   0  0  0  2  0     0
MatAssemblyEnd        47 1.0 6.1052e+01 1.1 0.00e+00 0.0 6.9e+02 1.8e+03 2.8e+02  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 0.0 8.9049e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCreateSubMat        2 1.0 1.2997e-01 1.0 0.00e+00 0.0 4.9e+01 2.9e+02 9.0e+01  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 0.0 4.1870e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             2 1.0 8.3524e-01 1.1 0.00e+00 0.0 7.6e+02 8.7e+03 2.4e+01  0  0  0  0  0   0  0  0  0  0     0
MatZeroEntries         3 1.0 3.3310e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLoad                1 1.0 2.3003e+01 1.0 0.00e+00 0.0 6.1e+01 4.4e+06 4.1e+01  0  0  0  1  0   0  0  0  1  0     0
MatAXPY                2 1.0 5.8209e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatTranspose           4 1.0 4.3042e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             2 1.0 4.1555e+00 1.0 1.96e+07 1.3 4.3e+02 6.5e+03 6.6e+01  0  0  0  0  0   0  0  0  0  0    31
MatMatMultSym          2 1.0 2.1865e+00 1.0 0.00e+00 0.0 3.6e+02 4.7e+03 6.2e+01  0  0  0  0  0   0  0  0  0  0     0
MatMatMultNum          2 1.0 1.9642e+00 1.0 1.96e+07 1.3 7.0e+01 1.6e+04 4.0e+00  0  0  0  0  0   0  0  0  0  0    65
MatPtAP                2 1.0 7.6737e+00 1.0 1.08e+08 1.3 7.6e+02 1.2e+04 8.0e+01  0  0  0  0  0   0  0  0  0  0   102
MatPtAPSymbolic        2 1.0 3.8572e+00 1.0 0.00e+00 0.0 4.4e+02 1.8e+04 3.0e+01  0  0  0  0  0   0  0  0  0  0     0
MatPtAPNumeric         2 1.0 3.8076e+00 1.0 1.08e+08 1.3 3.2e+02 3.3e+03 5.0e+01  0  0  0  0  0   0  0  0  0  0   206
MatTrnMatMult          1 1.0 3.2146e+02 1.0 3.78e+09 1.6 1.5e+02 6.7e+06 4.2e+01  1  0  0  3  0   1  0  0  3  0    69
MatTrnMatMultSym       1 1.0 7.1323e+01 1.0 0.00e+00 0.0 6.0e+01 2.5e+06 1.7e+01  0  0  0  0  0   0  0  0  0  0     0
MatTrnMatMultNum       1 1.0 2.5014e+02 1.0 3.78e+09 1.6 9.4e+01 9.4e+06 2.5e+01  0  0  0  2  0   0  0  0  2  0    89
MatGetLocalMat         7 1.0 5.9865e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          6 1.0 4.1407e-01 1.3 0.00e+00 0.0 4.9e+02 2.2e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatTranspose_SeqAIJ_FAST       4 1.0 4.0492e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               9 1.0 5.9289e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 5.3465e+04 1.0 1.19e+12 1.4 2.4e+06 1.6e+04 2.6e+05 99100100 96100  99100100 96100   141
KSPGMRESOrthog        40 1.0 7.0008e+00 7.5 3.09e+07 1.0 0.0e+00 0.0e+00 3.0e+02  0  0  0  0  0   0  0  0  0  0    35
PCGAMGGraph_AGG        2 1.0 1.3118e+01 1.0 1.96e+07 1.3 2.1e+02 6.7e+03 1.0e+02  0  0  0  0  0   0  0  0  0  0    10
PCGAMGCoarse_AGG       2 1.0 3.2407e+02 1.0 3.78e+09 1.6 1.1e+03 9.9e+05 8.4e+01  1  0  0  3  0   1  0  0  3  0    68
PCGAMGProl_AGG         2 1.0 8.8382e-01 1.8 0.00e+00 0.0 2.9e+02 2.1e+04 9.8e+01  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         2 1.0 1.4734e+01 1.0 2.36e+08 1.3 1.1e+03 1.2e+04 3.8e+02  0  0  0  0  0   0  0  0  0  0   107
GAMG: createProl       2 1.0 3.5259e+02 1.0 4.04e+09 1.6 2.7e+03 4.0e+05 6.6e+02  1  0  0  3  0   1  0  0  3  0    68
  Graph                4 1.0 1.3072e+01 1.0 1.96e+07 1.3 2.1e+02 6.7e+03 1.0e+02  0  0  0  0  0   0  0  0  0  0    10
  MIS/Agg              2 1.0 8.6046e-01 1.1 0.00e+00 0.0 7.6e+02 8.7e+03 2.4e+01  0  0  0  0  0   0  0  0  0  0     0
  SA: col data         2 1.0 2.2417e-01 1.0 0.00e+00 0.0 1.6e+02 3.2e+04 2.8e+01  0  0  0  0  0   0  0  0  0  0     0
  SA: frmProl0         2 1.0 2.3013e-01 1.0 0.00e+00 0.0 1.3e+02 5.6e+03 5.0e+01  0  0  0  0  0   0  0  0  0  0     0
  SA: smooth           2 1.0 4.8317e+00 1.0 2.02e+07 1.3 4.3e+02 6.5e+03 8.8e+01  0  0  0  0  0   0  0  0  0  0    28
GAMG: partLevel        2 1.0 7.9183e+00 1.0 1.08e+08 1.3 8.3e+02 1.1e+04 2.1e+02  0  0  0  0  0   0  0  0  0  0    99
  repartition          1 1.0 1.6927e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.3e+01  0  0  0  0  0   0  0  0  0  0     0
  Invert-Sort          1 1.0 4.2589e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.0e+00  0  0  0  0  0   0  0  0  0  0     0
  Move A               1 1.0 9.4407e-02 1.0 0.00e+00 0.0 3.5e+01 4.0e+02 4.7e+01  0  0  0  0  0   0  0  0  0  0     0
  Move P               1 1.0 5.1312e-02 1.0 0.00e+00 0.0 1.4e+01 4.0e+01 4.9e+01  0  0  0  0  0   0  0  0  0  0     0
PCSetUp                2 1.0 3.6084e+02 1.0 4.14e+09 1.6 3.5e+03 3.1e+05 9.8e+02  1  0  0  3  0   1  0  0  3  0    68
PCSetUpOnBlocks     4737 1.0 3.5424e-01 1.4 4.67e+04 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCApply             4737 1.0 5.0190e+04 1.0 1.09e+12 1.4 2.3e+06 1.4e+04 2.2e+05 93 91 96 83 85  93 91 96 83 85   138
SFSetGraph             2 1.0 8.9202e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetUp                2 1.0 3.6647e-01 1.1 0.00e+00 0.0 1.2e+02 5.4e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin          16 1.0 4.8668e-02 1.6 0.00e+00 0.0 6.4e+02 9.3e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastEnd            16 1.0 1.1840e-01 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     4              3         2544     0.
              Vector   135            135     16274000     0.
              Matrix    68             68    674831928     0.
      Matrix Coarsen     2              2         1288     0.
           Index Set    42             42       117744     0.
         Vec Scatter    15             15        20168     0.
       Krylov Solver     9              9       235736     0.
      Preconditioner     7              7         7236     0.
         PetscRandom     4              4         2680     0.
   Star Forest Graph     2              2         1760     0.
========================================================================================================================
Average time to get PetscTime(): 7.14209e-05
Average time for MPI_Barrier(): 0.000216455
Average time for zero size MPI_Send(): 0.000925563
#PETSc Option Table entries:
-ksp_type bcgs
-log_view
-malloc off
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 sizeof(PetscInt) 4
Configure options: PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
-----------------------------------------
Libraries compiled on 2019-01-10 10:35:56 on r02g03 
Machine characteristics: Linux-3.10.0-514.2.2.el7.x86_64-x86_64-with-centos-7.3.1611-Core
Using PETSc directory: /home/vef002/petsc
Using PETSc arch: linux-cumulus-debug
-----------------------------------------

Using C compiler: /usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc  -fPIC  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3  
Using Fortran compiler: /usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort  -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g    
-----------------------------------------

Using include paths: -I/home/vef002/petsc/include -I/home/vef002/petsc/linux-cumulus-debug/include
-----------------------------------------

Using C linker: /usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
Using Fortran linker: /usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
Using libraries: -Wl,-rpath,/home/vef002/petsc/linux-cumulus-debug/lib -L/home/vef002/petsc/linux-cumulus-debug/lib -lpetsc -Wl,-rpath,/home/vef002/petsc/linux-cumulus-debug/lib -L/home/vef002/petsc/linux-cumulus-debug/lib -Wl,-rpath,/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/lib -L/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/lib -Wl,-rpath,/usr/local/depot/gcc-7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -L/usr/local/depot/gcc-7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -Wl,-rpath,/usr/local/depot/gcc-7.3.0/lib64 -L/usr/local/depot/gcc-7.3.0/lib64 -Wl,-rpath,/usr/local/depot/gcc-7.3.0/lib -L/usr/local/depot/gcc-7.3.0/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu -lsuperlu_dist -lflapack -lfblas -lparmetis -lmetis -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ -ldl
-----------------------------------------



      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################

From tempohoper at gmail.com  Fri Jan 11 07:44:22 2019
From: tempohoper at gmail.com (Sal Am)
Date: Fri, 11 Jan 2019 13:44:22 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
Message-ID: <CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>

Using a larger problem set with 2B non-zero elements and a matrix of 25M x
25M I get the following error:
[4]PETSC ERROR:
------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
/lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
/lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable
line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
/lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428
/lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002
Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
--with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
--with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
--with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
--download-parmetis --download-metis --download-ptscotch
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=yes --download-scalapack --download-superlu
--download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was
written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
(matmatmult.c:790)
==9053==    by 0x5D106F8:
MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
(mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:

> Thank you Dave,
>
> I reconfigured PETSc with valgrind and debugging mode, I ran the code
> again with the following options:
> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
> gamg -log_view
> (as on the petsc website you linked)
>
> It finished solving using the iterative solver, but the resulting
> valgrind.log.%p files (all 8 corresponding to each processor) are all
> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>
>
> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com> wrote:
>
>>
>>
>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> I am not sure what is exactly is wrong as the error changes slightly
>>> every time I run it (without changing the parameters).
>>>
>>
>> This likely implies that you have a memory error in your code (a memory
>> leak would not cause this behaviour).
>> I strongly suggest you make sure your code is free of memory errors.
>> You can do this using valgrind. See here
>>
>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>
>> for an explanation of how to use valgrind.
>>
>>
>>> I have attached the first two run's errors and my code.
>>>
>>> Is there a memory leak somewhere? I have tried running it with
>>> -malloc_dump, but not getting anything printed out, however, when run with
>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>> way I see it, I have destroyed it where I see I no longer have use for it
>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>> (I have a print statement before solving starts that never prints).
>>>
>>> how I run it in the job script on 2 node with 32 processors using the
>>> clusters OpenMPI.
>>>
>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason
>>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>> -ksp_monitor -malloc_log -ksp_view
>>>
>>> the matrix:
>>> 2 122 821 366 (non-zero elements)
>>> 25 947 279 x 25 947 279
>>>
>>> Thanks and all the best
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/1cd6cf0c/attachment.html>

From chen2018 at purdue.edu  Thu Jan 10 21:56:44 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Fri, 11 Jan 2019 03:56:44 +0000
Subject: [petsc-users] PETSC for singular system
Message-ID: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>

Hello,


I am trying to use PETSC to solve a singular system like the following.

[cid:ef7b4672-623e-48d7-b9fa-6e8d4ed67b5f].

This equals inserting rows and columns at arbitrary place(Actually I even don't know whether it is singular or not. Sometimes some degree of freedom x_i may not participate in the global matrix)

[cid:0c347c3e-462d-4fe8-9718-273339c2c1f2]

I searched some discussion online and tried to remove the nullspace with the following code:

     call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
     call KSPSetOperators(ksp,Amat,Amat,ierr)
     call KSPSetFromOptions(ksp,ierr)
     call KSPSetType(ksp, KSPCG,ierr)
     call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, dummyVecs, nullspace, ierr)
     call MatSetNullSpace(Amat,nullspace,ierr)
     call MatNullSpaceDestroy(nullspace,ierr)
     call KSPSolve(ksp,bvec,xvec,ierr)

But it still gives me some error saying Zero pivot in LU factorization.
I am just wondering how MatSetNullSpace() can handle this?

Thanks


Yaxiong Chen,

School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/bb05442b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 2573 bytes
Desc: pastedImage.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/bb05442b/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3538 bytes
Desc: pastedImage.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/bb05442b/attachment-0003.png>

From jczhang at mcs.anl.gov  Fri Jan 11 09:52:43 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Fri, 11 Jan 2019 15:52:43 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
Message-ID: <CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>

I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/9ffef4e3/attachment.html>

From hzhang at mcs.anl.gov  Fri Jan 11 11:34:34 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Fri, 11 Jan 2019 17:34:34 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
Message-ID: <CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>

Add option '-mattransposematmult_via scalable'
Hong

On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/755d1ef6/attachment-0001.html>

From knepley at gmail.com  Fri Jan 11 13:27:58 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Jan 2019 14:27:58 -0500
Subject: [petsc-users] How Can I scatter a global vector to a local
 vector without using DMGlobaltoLocalBegin/DMGlobaltoLocalEnd?
In-Reply-To: <e88b97f0457316cbbefb049c6344b9dbc6580f80.camel@mail.nwpu.edu.cn>
References: <e88b97f0457316cbbefb049c6344b9dbc6580f80.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4G=8HgQaDosb_GzGb6jaeMC3ptbgNOHWs3raROgYHr_yVQ@mail.gmail.com>

On Fri, Jan 11, 2019 at 3:35 AM leejearl via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi, all Petscer:
>   I am very sorry for disturbing you for my questions.


I like your questions.


> I have a
> problems as follow.
>   1. A DMPlex object dm is created.
>   2. A global vector and a local vector are created using the following
> routines DMCreateGlobalVector and DMCreateLocalVector.
>   3. I have initialed global Vector. Because the local vector have some
> overlap cells, I want to scatter the data from global to local.
>   4. If I don't use the routines
> DMGlobaltoLocalBegin/DMGlobaltoLocalEnd, how can I scatter it?


To understand the question better, why would you not use DMGlobalToLocal()?


> I have
> scatter the vectors using VecScatterBegin and VecScatterEnd, and the IS
> is created using the localId and the glolbalId (Obtained by
> DMPlexGetCellNumbering). I got a wrong result.
>

DMPlexGetCellNumbering() gives you are _arbitrary_ numbering of the cells,
not the
global numbering of unknowns.


>   Can anyone give me some advices, and does it have some to do with the
> natural ids?
>

1) You can get the local ids from the local Section (DMGetSection), and the
global ids
from the global Section (DMGetGlobalSection). You loop over all the points
in the mesh
and get the offset/size pairs. However, this is error-prone (since you can
have constraints)
and probably unnecessary.

2) If you look at the code for DMGlobalToLocalBegin():

https://bitbucket.org/petsc/petsc/src/650136806607101f8eb495880e9efa33f2f75729/src/dm/interface/dm.c#lines-2201

you see it just call PetscSFBcastBegin(), and there is a similar BcastEnd()
in GlobalToLocalEnd(). Thus all
we are doing is using the SF directly. That is the same as using a Scatter.

  Thanks,

   Matt


>   Any helps are approciated. Thanks.
>
> leejearl
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/ffa96a31/attachment.html>

From knepley at gmail.com  Fri Jan 11 13:30:18 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Jan 2019 14:30:18 -0500
Subject: [petsc-users] PETSC for singular system
In-Reply-To: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>

On Fri, Jan 11, 2019 at 10:04 AM Yaxiong Chen via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
>
> I am trying to use PETSC to solve a singular system like the following.
>
> .
>
> This equals inserting rows and columns at arbitrary place(Actually I
> even don't know whether it is singular or not. Sometimes some degree of
> freedom x_i may not participate in the global matrix)
>
>
> I searched some discussion online and tried to remove the nullspace with
> the following code:
>
>      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>      call KSPSetOperators(ksp,Amat,Amat,ierr)
>      call KSPSetFromOptions(ksp,ierr)
>      call KSPSetType(ksp, KSPCG,ierr)
>      call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, dummyVecs,
> nullspace, ierr)
>      call MatSetNullSpace(Amat,nullspace,ierr)
>      call MatNullSpaceDestroy(nullspace,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>
> But it still gives me some error saying Zero pivot in LU factorization.
> I am just wondering how MatSetNullSpace() can handle this?
>

If you give the nullspace, the Kryloc method will work. However, you still
cannot use factorization. So

1) -ksp_type gmres -pc_type jacobi will solve your system

2) -ksp_type gmres -pc_type svd will solve your system

Saying anything more general than that is really not possible.

  Thanks,

    Matt


> Thanks
>
> Yaxiong Chen,
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/2b1ded17/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 2573 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/2b1ded17/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3538 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190111/2b1ded17/attachment-0001.png>

From bhatiamanav at gmail.com  Sat Jan 12 00:03:43 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Fri, 11 Jan 2019 22:03:43 -0800
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
Message-ID: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>

Hi,

   I am interested in setting up a high-order DG FEM solution with PETSc on an octree mesh with adaptivity and geometric multigrid preconditioning. Is there an example that demonstrates this with either DMForest or DMPlex. 
  
Thanks,
Manav



From bhatiamanav at gmail.com  Sat Jan 12 01:46:42 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Fri, 11 Jan 2019 23:46:42 -0800
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
Message-ID: <503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>

While looking through some examples I see that both PetscDSSetJacobian and DMPlexSNESComputeJacobianFEM are being used to set the functions to evaluate the Jacobian contributions on each rank. 

This is my understanding after studying these (please correct if I am wrong): 

? The PetscDSSetJacobian (and its corresponding residual routine) does not seem to have the concept of an element. Instead, it provides the derivatives of solution at a point and requires that the coefficients for each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the background. 

? DMPlexSNESComputeJacobianFEM is a method that calls PetscDSSetJacobian to compute the Jacobian. 


If this is correct, I am a bit confused about the following: 

? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to compute the convective flux at the interface of two elements. However, I don?t see how the Jacobian contribution of this flux can be computed and added to the system Jacobian. 

? A more conventional FEM assembly iterates over elements, then over the quadrature points, where the shape functions and their derivatives are initialized and used for computation of residual and Jacobian. Is it possible to follow this procedure in PETSc with DMPlex/DMForest and still use multigrid? Is there an example that demonstrates this? 

? Is there some form of an index for elements that exist on the local processor that can be used to iterate on local elements? 


I have tried to look, but could not find a document that outlines these concepts. Is there one that exists and I have missed it? 

I would greatly appreciate some direction. 

Regards,
Manav




> On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Hi,
> 
>   I am interested in setting up a high-order DG FEM solution with PETSc on an octree mesh with adaptivity and geometric multigrid preconditioning. Is there an example that demonstrates this with either DMForest or DMPlex. 
> 
> Thanks,
> Manav
> 
> 


From bhatiamanav at gmail.com  Sat Jan 12 17:58:05 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Sat, 12 Jan 2019 18:58:05 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
Message-ID: <0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>

I have been studying the source code and the manual entries in the documentation and have a better understanding of setting up a high-order DG analysis. 

I have been able to identify the routines that set the order of the space and its continuity (CG/DG). 

The following is still unclear and I would appreciate some guidance: 

? A implicit DG solve of the inviscid Euler equations would require Jacobian contributions from the Riemann Solver at element interfaces. I do not see the ability to define Jacobian contribution in PetscDSSetRiemannSolver. What would be the best approach to do this? 

? It appears that the only two choices for polynomials are Lagrange and ?simple?. I am not clear what the ?simple? basis implies. Are these just powers of x, y and z? 

? Are Legendre polynomials available elsewhere in PETSc? I saw some mention of it in TS, but not in FE. 

Thanks,
Manav



> On Jan 12, 2019, at 2:46 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> While looking through some examples I see that both PetscDSSetJacobian and DMPlexSNESComputeJacobianFEM are being used to set the functions to evaluate the Jacobian contributions on each rank. 
> 
> This is my understanding after studying these (please correct if I am wrong): 
> 
> ? The PetscDSSetJacobian (and its corresponding residual routine) does not seem to have the concept of an element. Instead, it provides the derivatives of solution at a point and requires that the coefficients for each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the background. 
> 
> ? DMPlexSNESComputeJacobianFEM is a method that calls PetscDSSetJacobian to compute the Jacobian. 
> 
> 
> If this is correct, I am a bit confused about the following: 
> 
> ? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to compute the convective flux at the interface of two elements. However, I don?t see how the Jacobian contribution of this flux can be computed and added to the system Jacobian. 
> 
> ? A more conventional FEM assembly iterates over elements, then over the quadrature points, where the shape functions and their derivatives are initialized and used for computation of residual and Jacobian. Is it possible to follow this procedure in PETSc with DMPlex/DMForest and still use multigrid? Is there an example that demonstrates this? 
> 
> ? Is there some form of an index for elements that exist on the local processor that can be used to iterate on local elements? 
> 
> 
> I have tried to look, but could not find a document that outlines these concepts. Is there one that exists and I have missed it? 
> 
> I would greatly appreciate some direction. 
> 
> Regards,
> Manav
> 
> 
> 
> 
>> On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Hi,
>> 
>>  I am interested in setting up a high-order DG FEM solution with PETSc on an octree mesh with adaptivity and geometric multigrid preconditioning. Is there an example that demonstrates this with either DMForest or DMPlex. 
>> 
>> Thanks,
>> Manav
>> 
>> 
> 


From mfadams at lbl.gov  Sun Jan 13 08:20:35 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Sun, 13 Jan 2019 09:20:35 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
	<0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
Message-ID: <CADOhEh4zYvRZ_WazfgQ8n3RwGk0TC_1P7K+S=3jW-9DT559vwg@mail.gmail.com>

On Sat, Jan 12, 2019 at 6:59 PM Manav Bhatia via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I have been studying the source code and the manual entries in the
> documentation and have a better understanding of setting up a high-order DG
> analysis.
>
> I have been able to identify the routines that set the order of the space
> and its continuity (CG/DG).
>
> The following is still unclear and I would appreciate some guidance:
>
> ? A implicit DG solve of the inviscid Euler equations would require
> Jacobian contributions from the Riemann Solver at element interfaces. I do
> not see the ability to define Jacobian contribution in
> PetscDSSetRiemannSolver. What would be the best approach to do this?
>

See
https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex11.c.html

The Riemann solver computes the flux and the Jacobian is assembled in PETSc.


>
> ? It appears that the only two choices for polynomials are Lagrange and
> ?simple?. I am not clear what the ?simple? basis implies. Are these just
> powers of x, y and z?
>
> ? Are Legendre polynomials available elsewhere in PETSc? I saw some
> mention of it in TS, but not in FE.
>
> Thanks,
> Manav
>
>
>
> > On Jan 12, 2019, at 2:46 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> >
> > While looking through some examples I see that both PetscDSSetJacobian
> and DMPlexSNESComputeJacobianFEM are being used to set the functions to
> evaluate the Jacobian contributions on each rank.
> >
> > This is my understanding after studying these (please correct if I am
> wrong):
> >
> > ? The PetscDSSetJacobian (and its corresponding residual routine) does
> not seem to have the concept of an element. Instead, it provides the
> derivatives of solution at a point and requires that the coefficients for
> each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the
> background.
> >
> > ? DMPlexSNESComputeJacobianFEM is a method that calls PetscDSSetJacobian
> to compute the Jacobian.
> >
> >
> > If this is correct, I am a bit confused about the following:
> >
> > ? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to
> compute the convective flux at the interface of two elements. However, I
> don?t see how the Jacobian contribution of this flux can be computed and
> added to the system Jacobian.
> >
> > ? A more conventional FEM assembly iterates over elements, then over the
> quadrature points, where the shape functions and their derivatives are
> initialized and used for computation of residual and Jacobian. Is it
> possible to follow this procedure in PETSc with DMPlex/DMForest and still
> use multigrid? Is there an example that demonstrates this?
> >
> > ? Is there some form of an index for elements that exist on the local
> processor that can be used to iterate on local elements?
> >
> >
> > I have tried to look, but could not find a document that outlines these
> concepts. Is there one that exists and I have missed it?
> >
> > I would greatly appreciate some direction.
> >
> > Regards,
> > Manav
> >
> >
> >
> >
> >> On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >>  I am interested in setting up a high-order DG FEM solution with PETSc
> on an octree mesh with adaptivity and geometric multigrid preconditioning.
> Is there an example that demonstrates this with either DMForest or DMPlex.
> >>
> >> Thanks,
> >> Manav
> >>
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/b612c8bb/attachment.html>

From jed at jedbrown.org  Sun Jan 13 08:40:33 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 13 Jan 2019 07:40:33 -0700
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <CADOhEh4zYvRZ_WazfgQ8n3RwGk0TC_1P7K+S=3jW-9DT559vwg@mail.gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
	<0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
	<CADOhEh4zYvRZ_WazfgQ8n3RwGk0TC_1P7K+S=3jW-9DT559vwg@mail.gmail.com>
Message-ID: <87sgxwfwny.fsf@jedbrown.org>

Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:

>> The following is still unclear and I would appreciate some guidance:
>>
>> ? A implicit DG solve of the inviscid Euler equations would require
>> Jacobian contributions from the Riemann Solver at element interfaces. I do
>> not see the ability to define Jacobian contribution in
>> PetscDSSetRiemannSolver. What would be the best approach to do this?
>>
>
> See
> https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex11.c.html
>
> The Riemann solver computes the flux and the Jacobian is assembled in PETSc.

If you run this example with an implicit integrator, it uses FD coloring
to compute the Jacobian.  Manav wants an analytic Jacobian.  I was
hoping Matt could answer this; I haven't used Plex in that way.

From valerio.barnabei at gmail.com  Sun Jan 13 08:57:44 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Sun, 13 Jan 2019 15:57:44 +0100
Subject: [petsc-users] KSP solve time and iterations
Message-ID: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>

 Hello everybody,

I have some doubts about the parallel solution of a simple FEM ksp problem.
I searched in the mailing list but the information I found are not helping
me.
I need to control manually the parallel assemble of the main matrix, but I
would like to use PetSC for the solution of the final linear system K*U=F.
The algorithm I am using, loads an already decomposed FEM domain, and each
process contributes to the global matrix assembly using its portion of
domain decomposition (made in pre-processing). In the element loop, at each
element I load the element submatrix in the global matrix via MatSetValue
function. Then, out of the element loop, I write
/*
...matrix preallocation
...calculate element local stiffness matrix (local in a FEM sense and local
in a domain decomposition sense)
MatSetValue (ADD)
*/
and then
ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
ierr=VecAssemblyEnd(F);CHKERRQ(ierr);

This part scales perfectly and is sufficiently fast.
My K matrix is a sort of banded diagonal matrix: my preallocation is not
perfect, but it works at this stage of my work.
Finally I solve the system with the ksp logic:

ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI, the
behaviour is almost identical, the iteration number lowers but is still
really high
ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
ierr=KSPSolve(ksp,F,sol);

When I run the code I am noticing some strange behaviours. Indeed, the
numerical solution appear right, and the code scales if I make a simple
strong-scaling test. However, the ksp_log and log_view (see below, logs for
a 160k quad linear element uniform 2dmesh) shows me a huge number of
iterations, and the time spent in "solve" function is very high.
Furthermore, the more I increase the problem size keeping a constant load
per process (weak scaling) the more iterations seems to be needed for
convergence. In general, when increasing the problem size (i.e. 16M
elements) the convergence is not achieved before the maximum iteration
number is achieved.

Question 1: Am I doing it right? Or am I making some trouble with the MPI
communication management? (i'm aware that my domain decomposition does not
match the petsc matrix decomposition, I'm expecting it to cripple my
algorithm a bit)
Question 2: Why the solver takes so much time and so many iterations? the
problem is really simple, and the solution appears correct when plotted. Am
I ill conditioning the system?
Thanks in advance for your help. I can add any further information if
needed.

Valerio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/07156c74/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_weak16.out
Type: application/octet-stream
Size: 11143 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/07156c74/attachment-0001.obj>

From stefano.zampini at gmail.com  Sun Jan 13 09:08:36 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Sun, 13 Jan 2019 18:08:36 +0300
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
Message-ID: <CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>

What problem are you trying to solve?
For standard elliptic problems (e.g. poisson or elasticity) you should use
a preconditioner that scales, for example PCGAMG or BoomerAMG from HYPRE
(PCHYPRE, if you configured PETSc with support for HYPRE via
--download-hypre)
Alternatively, since you are using FEM with local Assembly, you can use
PCBDDC, but this requires few extra calls

- call MatSetType(K,MATIS)
- provide an ISLocalToGlobalMapping of local to global degrees of freedom
before you set preallocation via MatSetLocalToGlobalMapping (if you provide
this call, then you can use MatSetValuesLocal for both AIJ amd MATIS types)



Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

> Hello everybody,
>
> I have some doubts about the parallel solution of a simple FEM ksp
> problem. I searched in the mailing list but the information I found are not
> helping me.
> I need to control manually the parallel assemble of the main matrix, but I
> would like to use PetSC for the solution of the final linear system K*U=F.
> The algorithm I am using, loads an already decomposed FEM domain, and each
> process contributes to the global matrix assembly using its portion of
> domain decomposition (made in pre-processing). In the element loop, at each
> element I load the element submatrix in the global matrix via MatSetValue
> function. Then, out of the element loop, I write
> /*
> ...matrix preallocation
> ...calculate element local stiffness matrix (local in a FEM sense and
> local in a domain decomposition sense)
> MatSetValue (ADD)
> */
> and then
> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>
> This part scales perfectly and is sufficiently fast.
> My K matrix is a sort of banded diagonal matrix: my preallocation is not
> perfect, but it works at this stage of my work.
> Finally I solve the system with the ksp logic:
>
> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI, the
> behaviour is almost identical, the iteration number lowers but is still
> really high
>
> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
> ierr=KSPSolve(ksp,F,sol);
>
> When I run the code I am noticing some strange behaviours. Indeed, the
> numerical solution appear right, and the code scales if I make a simple
> strong-scaling test. However, the ksp_log and log_view (see below, logs for
> a 160k quad linear element uniform 2dmesh) shows me a huge number of
> iterations, and the time spent in "solve" function is very high.
> Furthermore, the more I increase the problem size keeping a constant load
> per process (weak scaling) the more iterations seems to be needed for
> convergence. In general, when increasing the problem size (i.e. 16M
> elements) the convergence is not achieved before the maximum iteration
> number is achieved.
>
> Question 1: Am I doing it right? Or am I making some trouble with the MPI
> communication management? (i'm aware that my domain decomposition does not
> match the petsc matrix decomposition, I'm expecting it to cripple my
> algorithm a bit)
> Question 2: Why the solver takes so much time and so many iterations? the
> problem is really simple, and the solution appears correct when plotted. Am
> I ill conditioning the system?
> Thanks in advance for your help. I can add any further information if
> needed.
>
> Valerio
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/1a23f79b/attachment.html>

From evanum at gmail.com  Sun Jan 13 11:07:54 2019
From: evanum at gmail.com (Evan Um)
Date: Sun, 13 Jan 2019 09:07:54 -0800
Subject: [petsc-users] Using real and complex matrices together
Message-ID: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>

Hi PETSC users,

Inside my codes, I have both complex and real matrices. To solve complex
matrices, I build and use complex version of PETSC. In this case, is there
any way to use real matrices too? Do you have such a PETSC code example?
Until now, I have used complex matrices to represent real matrices by
setting imaginary components of the real matrices to zero. Real value
vectors are also realized by setting imaginary components to zero.

In advance, thanks for your comments.

Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/65141dde/attachment.html>

From jed at jedbrown.org  Sun Jan 13 11:47:54 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 13 Jan 2019 10:47:54 -0700
Subject: [petsc-users] Using real and complex matrices together
In-Reply-To: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>
References: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>
Message-ID: <87pnt0fnzp.fsf@jedbrown.org>

It is not supported, sorry.  Some users do it with careful linking via
multiple shared libraries, but that is fragile and doesn't interoperate.

Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi PETSC users,
>
> Inside my codes, I have both complex and real matrices. To solve complex
> matrices, I build and use complex version of PETSC. In this case, is there
> any way to use real matrices too? Do you have such a PETSC code example?
> Until now, I have used complex matrices to represent real matrices by
> setting imaginary components of the real matrices to zero. Real value
> vectors are also realized by setting imaginary components to zero.
>
> In advance, thanks for your comments.
>
> Evan

From mfadams at lbl.gov  Sun Jan 13 13:56:10 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Sun, 13 Jan 2019 14:56:10 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <87sgxwfwny.fsf@jedbrown.org>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
	<0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
	<CADOhEh4zYvRZ_WazfgQ8n3RwGk0TC_1P7K+S=3jW-9DT559vwg@mail.gmail.com>
	<87sgxwfwny.fsf@jedbrown.org>
Message-ID: <CADOhEh4qwDmh2ciQNxYgRX6z6CAGzN4x1sP74hySFUcRA=W5EA@mail.gmail.com>

>
>
>
> > The Riemann solver computes the flux and the Jacobian is assembled in
> PETSc.
>
> If you run this example with an implicit integrator, it uses FD coloring
> to compute the Jacobian.  Manav wants an analytic Jacobian.  I was
> hoping Matt could answer this; I haven't used Plex in that way.
>

Oh right. That test is explicit.

Matt is on vacation but he is doing email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/93270542/attachment.html>

From knepley at gmail.com  Sun Jan 13 13:57:27 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 13 Jan 2019 14:57:27 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
Message-ID: <CAMYG4Gk-F3aSVUh1Mi3QLBxfeToikjFfy=iaUJbX-EfM1WEv1g@mail.gmail.com>

On Sat, Jan 12, 2019 at 2:47 AM Manav Bhatia via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> While looking through some examples I see that both PetscDSSetJacobian and
> DMPlexSNESComputeJacobianFEM are being used to set the functions to
> evaluate the Jacobian contributions on each rank.
>

Let me say first that we do not support DG. We have plans to do this, but
it does not currently work.


> This is my understanding after studying these (please correct if I am
> wrong):
>
> ? The PetscDSSetJacobian (and its corresponding residual routine) does not
> seem to have the concept of an element. Instead, it provides the
> derivatives of solution at a point and requires that the coefficients for
> each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the
> background.
>

Yes, this is true. Here is the idea: DS abstracts the problem itself,
without regard to discretization.The division is not exactly what you
expect currently, but I think it is the correct one. You are solving the
weak form of the problem, which is definitely different than the strong
form, so the DS assumes a weak form for the specification, but independent
of the bases involved.


> ? DMPlexSNESComputeJacobianFEM is a method that calls PetscDSSetJacobian
> to compute the Jacobian.
>

Yes.


> If this is correct, I am a bit confused about the following:
>
> ? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to
> compute the convective flux at the interface of two elements. However, I
> don?t see how the Jacobian contribution of this flux can be computed and
> added to the system Jacobian.
>

That is really only designed for FV. You are right that we could repurpose
it to support DG. That was not how I was going to proceed. Instead I have a
branch that fixes boundary integration so that all interface integrals get
2-sided information. This would allow all the facets integrals that you
want with proper transforms, etc. However, I would also have to put in
support for geometric quantities needed for DG penalization. That is
simple, I know, but everything takes time.


> ? A more conventional FEM assembly iterates over elements, then over the
> quadrature points, where the shape functions and their derivatives are
> initialized and used for computation of residual and Jacobian. Is it
> possible to follow this procedure in PETSc with DMPlex/DMForest and still
> use multigrid? Is there an example that demonstrates this?
>

Yes, that is how all the FEM examples work. For instance, you can look at
SNES ex12 or ex17 or ex62 or ex77. They iterates over batches
of cells, then projectbasis function to the quad points, evaluate user
point functions on the quad points, and then take the inner product with
test functions. After the element matrix is computed, it is set into the
global Jacobian.


> ? Is there some form of an index for elements that exist on the local
> processor that can be used to iterate on local elements?
>

We just use local point number to iterate over cells.


> I have tried to look, but could not find a document that outlines these
> concepts. Is there one that exists and I have missed it?
>

There ia a manual chapter on the low-level organization, but nothing on the
traversal because I am still experimenting.

  Thanks,

   Matt


> I would greatly appreciate some direction.
>
> Regards,
> Manav
>
>
>
>
> > On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com>
> wrote:
> >
> > Hi,
> >
> >   I am interested in setting up a high-order DG FEM solution with PETSc
> on an octree mesh with adaptivity and geometric multigrid preconditioning.
> Is there an example that demonstrates this with either DMForest or DMPlex.
> >
> > Thanks,
> > Manav
> >
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/a26c0106/attachment.html>

From devyani at gatech.edu  Sat Jan 12 19:39:22 2019
From: devyani at gatech.edu (Choudhary, Devyani D)
Date: Sun, 13 Jan 2019 01:39:22 +0000
Subject: [petsc-users] C++ compilation error
Message-ID: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>

Hi,


I am trying to make a simple hello world script using a makefile that includes petsc, and am getting the error

"g++: error: unrecognized command line option ?-wd1572?"

I am not sure how to fix it.

Please let me know if you have some ideas on how to fix this.


Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/bf459753/attachment.html>

From knepley at gmail.com  Sun Jan 13 14:01:59 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 13 Jan 2019 15:01:59 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
	<0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
Message-ID: <CAMYG4G=ETOyWiVJ-sABZypZs74D42NO4My=DXmf6V4H0KtWZPw@mail.gmail.com>

On Sat, Jan 12, 2019 at 6:59 PM Manav Bhatia via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I have been studying the source code and the manual entries in the
> documentation and have a better understanding of setting up a high-order DG
> analysis.
>
> I have been able to identify the routines that set the order of the space
> and its continuity (CG/DG).
>
> The following is still unclear and I would appreciate some guidance:
>
> ? A implicit DG solve of the inviscid Euler equations would require
> Jacobian contributions from the Riemann Solver at element interfaces. I do
> not see the ability to define Jacobian contribution in
> PetscDSSetRiemannSolver. What would be the best approach to do this?
>

You would set a BdResidual/BdJacobian function. You need my branch that
makes these integrals 2-sided. However, you would also probably need to
make an aux field with the geometric quantities you need. I can help you do
that. Then we could roll that up and put it in the library.


> ? It appears that the only two choices for polynomials are Lagrange and
> ?simple?. I am not clear what the ?simple? basis implies. Are these just
> powers of x, y and z?
>

Correct. We would have to stick in another polynomial space. This is
actually pretty easy. We already have code for Legendre in the
PetscQuadrature code, and we should also probably add Chebyshev. In
addition, you probably want GLL quadrature. Some frustrating
people put GLL code in TS, which should be pulled out and made a
PetscQuadrature. You can see that this is some coding to do, but
none of it is design/hard, it just needs to be done.

  Thanks,

    Matt


> ? Are Legendre polynomials available elsewhere in PETSc? I saw some
> mention of it in TS, but not in FE.
>
> Thanks,
> Manav
>
>
>
> > On Jan 12, 2019, at 2:46 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> >
> > While looking through some examples I see that both PetscDSSetJacobian
> and DMPlexSNESComputeJacobianFEM are being used to set the functions to
> evaluate the Jacobian contributions on each rank.
> >
> > This is my understanding after studying these (please correct if I am
> wrong):
> >
> > ? The PetscDSSetJacobian (and its corresponding residual routine) does
> not seem to have the concept of an element. Instead, it provides the
> derivatives of solution at a point and requires that the coefficients for
> each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the
> background.
> >
> > ? DMPlexSNESComputeJacobianFEM is a method that calls PetscDSSetJacobian
> to compute the Jacobian.
> >
> >
> > If this is correct, I am a bit confused about the following:
> >
> > ? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to
> compute the convective flux at the interface of two elements. However, I
> don?t see how the Jacobian contribution of this flux can be computed and
> added to the system Jacobian.
> >
> > ? A more conventional FEM assembly iterates over elements, then over the
> quadrature points, where the shape functions and their derivatives are
> initialized and used for computation of residual and Jacobian. Is it
> possible to follow this procedure in PETSc with DMPlex/DMForest and still
> use multigrid? Is there an example that demonstrates this?
> >
> > ? Is there some form of an index for elements that exist on the local
> processor that can be used to iterate on local elements?
> >
> >
> > I have tried to look, but could not find a document that outlines these
> concepts. Is there one that exists and I have missed it?
> >
> > I would greatly appreciate some direction.
> >
> > Regards,
> > Manav
> >
> >
> >
> >
> >> On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >>  I am interested in setting up a high-order DG FEM solution with PETSc
> on an octree mesh with adaptivity and geometric multigrid preconditioning.
> Is there an example that demonstrates this with either DMForest or DMPlex.
> >>
> >> Thanks,
> >> Manav
> >>
> >>
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/bda7cfc0/attachment-0001.html>

From balay at mcs.anl.gov  Sun Jan 13 14:07:58 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sun, 13 Jan 2019 20:07:58 +0000
Subject: [petsc-users] C++ compilation error
In-Reply-To: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
References: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
Message-ID: <alpine.LFD.2.21.1901131402190.10937@sb>

Presumably PETSc is buit with intel compilers - but somehow gcc/g++ is
getting used via your makefile.

Do you get this error when you use PETSc example with a petsc
makefile? Perhaps you need to format your makefile using petsc
makefile format. For ex: check
src/tao/leastsquares/examples/tutorials/makefile

Satish

On Sun, 13 Jan 2019, Choudhary, Devyani D via petsc-users wrote:

> Hi,
> 
> 
> I am trying to make a simple hello world script using a makefile that includes petsc, and am getting the error
> 
> "g++: error: unrecognized command line option ?-wd1572?"
> 
> I am not sure how to fix it.
> 
> Please let me know if you have some ideas on how to fix this.
> 
> 
> Thank you
> 

From knepley at gmail.com  Sun Jan 13 14:07:37 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 13 Jan 2019 15:07:37 -0500
Subject: [petsc-users] C++ compilation error
In-Reply-To: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
References: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
Message-ID: <CAMYG4G=i6z1MawMhU-89wyoYFg9ULgSX5iUpzX6KxhdsMNh3HQ@mail.gmail.com>

On Sun, Jan 13, 2019 at 2:58 PM Choudhary, Devyani D via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
>
> I am trying to make a simple hello world script using a makefile that
> includes petsc, and am getting the error
>
> "g++: error: unrecognized command line option ?-wd1572?"
>
> I am not sure how to fix it.
>
> Please let me know if you have some ideas on how to fix this.
>

Can you run a PETSc example? Instructions for this are in the manual.

  Thanks,

   Matt


> Thank you
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/0364fecd/attachment.html>

From devyani at gatech.edu  Sun Jan 13 14:27:05 2019
From: devyani at gatech.edu (Choudhary, Devyani D)
Date: Sun, 13 Jan 2019 20:27:05 +0000
Subject: [petsc-users] C++ compilation error
In-Reply-To: <alpine.LFD.2.21.1901131402190.10937@sb>
References: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>,
	<alpine.LFD.2.21.1901131402190.10937@sb>
Message-ID: <BL0PR07MB52524BE1BBB050F5E2450AB0D4870@BL0PR07MB5252.namprd07.prod.outlook.com>

Thank you for the response Satish, but I am still encountering the same issue.

Following is my makefile:


PETSC_DIR=/usr/local/pacerepov1/petsc/3.8.3/mvapich2-2.1/intel-15.0/opt/lib/petsc/

include ${PETSC_DIR}/conf/variables
include ${PETSC_DIR}/conf/rules
include ${PETSC_DIR}/conf/test

hello: hello.o  chkopts
        -${CLINKER} -o hello hello.o  ${PETSC_LIB}
        ${RM} hello.o

It is in the format of petsc examples. I am not sure why I am still getting the error.

Devyani



________________________________
From: Balay, Satish <balay at mcs.anl.gov>
Sent: Sunday, January 13, 2019 3:07:58 PM
To: Choudhary, Devyani D
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] C++ compilation error

Presumably PETSc is buit with intel compilers - but somehow gcc/g++ is
getting used via your makefile.

Do you get this error when you use PETSc example with a petsc
makefile? Perhaps you need to format your makefile using petsc
makefile format. For ex: check
src/tao/leastsquares/examples/tutorials/makefile

Satish

On Sun, 13 Jan 2019, Choudhary, Devyani D via petsc-users wrote:

> Hi,
>
>
> I am trying to make a simple hello world script using a makefile that includes petsc, and am getting the error
>
> "g++: error: unrecognized command line option ?-wd1572?"
>
> I am not sure how to fix it.
>
> Please let me know if you have some ideas on how to fix this.
>
>
> Thank you
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/ce5c25b7/attachment.html>

From mpovolot at purdue.edu  Sun Jan 13 15:13:32 2019
From: mpovolot at purdue.edu (Povolotskyi, Mykhailo)
Date: Sun, 13 Jan 2019 21:13:32 +0000
Subject: [petsc-users] Using real and complex matrices together
In-Reply-To: <87pnt0fnzp.fsf@jedbrown.org>
References: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>,
	<87pnt0fnzp.fsf@jedbrown.org>
Message-ID: <f760a47783c9424e99b9cc2135222a52@wppexc05.purdue.lcl>

Hi Evan,

I?m one of those users, who has multiple shared libraries.

It is quite stable since I create wrappers around the PETSc functions and hide the native PETSc symbols.

But, yes, the interoperations are not possible.

Michael.



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10



From: Jed Brown via petsc-users<mailto:petsc-users at mcs.anl.gov>
Sent: Sunday, January 13, 2019 12:48 PM
To: Evan Um<mailto:evanum at gmail.com>; petsc-users<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using real and complex matrices together



It is not supported, sorry.  Some users do it with careful linking via
multiple shared libraries, but that is fragile and doesn't interoperate.

Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi PETSC users,
>
> Inside my codes, I have both complex and real matrices. To solve complex
> matrices, I build and use complex version of PETSC. In this case, is there
> any way to use real matrices too? Do you have such a PETSC code example?
> Until now, I have used complex matrices to represent real matrices by
> setting imaginary components of the real matrices to zero. Real value
> vectors are also realized by setting imaginary components to zero.
>
> In advance, thanks for your comments.
>
> Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/94ea0905/attachment.html>

From balay at mcs.anl.gov  Sun Jan 13 16:18:22 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sun, 13 Jan 2019 22:18:22 +0000
Subject: [petsc-users] C++ compilation error
In-Reply-To: <BL0PR07MB52524BE1BBB050F5E2450AB0D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
References: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>,
	<alpine.LFD.2.21.1901131402190.10937@sb>,
	<BL0PR07MB52524BE1BBB050F5E2450AB0D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
Message-ID: <BN7PR09MB262877C88370E3424C707B64ED870@BN7PR09MB2628.namprd09.prod.outlook.com>

Do you get this error with any petsc example ? say src/ksp/ksp/examples /tutorials/ex2 ?

Also, copy paste the complete output from make ? when building example or your Test code.

Satish

________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Choudhary, Devyani D via petsc-users <petsc-users at mcs.anl.gov>
Sent: Sunday, January 13, 2019 2:27:05 PM
To: petsc-users
Subject: Re: [petsc-users] C++ compilation error


Thank you for the response Satish, but I am still encountering the same issue.

Following is my makefile:


PETSC_DIR=/usr/local/pacerepov1/petsc/3.8.3/mvapich2-2.1/intel-15.0/opt/lib/petsc/

include ${PETSC_DIR}/conf/variables
include ${PETSC_DIR}/conf/rules
include ${PETSC_DIR}/conf/test

hello: hello.o  chkopts
        -${CLINKER} -o hello hello.o  ${PETSC_LIB}
        ${RM} hello.o

It is in the format of petsc examples. I am not sure why I am still getting the error.

Devyani



________________________________
From: Balay, Satish <balay at mcs.anl.gov>
Sent: Sunday, January 13, 2019 3:07:58 PM
To: Choudhary, Devyani D
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] C++ compilation error

Presumably PETSc is buit with intel compilers - but somehow gcc/g++ is
getting used via your makefile.

Do you get this error when you use PETSc example with a petsc
makefile? Perhaps you need to format your makefile using petsc
makefile format. For ex: check
src/tao/leastsquares/examples/tutorials/makefile

Satish

On Sun, 13 Jan 2019, Choudhary, Devyani D via petsc-users wrote:

> Hi,
>
>
> I am trying to make a simple hello world script using a makefile that includes petsc, and am getting the error
>
> "g++: error: unrecognized command line option ?-wd1572?"
>
> I am not sure how to fix it.
>
> Please let me know if you have some ideas on how to fix this.
>
>
> Thank you
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/15781808/attachment-0001.html>

From devyani at gatech.edu  Sun Jan 13 17:06:26 2019
From: devyani at gatech.edu (Choudhary, Devyani D)
Date: Sun, 13 Jan 2019 23:06:26 +0000
Subject: [petsc-users] Operand '<' error
Message-ID: <BL0PR07MB52521620497707A93398BDA3D4870@BL0PR07MB5252.namprd07.prod.outlook.com>

Hi,


I am trying to compare two PetscScalar values, but am getting this error:


error: no operator "<" matches these operands
operand types are: PetscScalar < PetscScalar

Please do share if you know how to resolve this error.

Sincerely,
Devyani

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190113/de5e4d76/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan 13 17:11:04 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 13 Jan 2019 23:11:04 +0000
Subject: [petsc-users] Operand '<' error
In-Reply-To: <BL0PR07MB52521620497707A93398BDA3D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
References: <BL0PR07MB52521620497707A93398BDA3D4870@BL0PR07MB5252.namprd07.prod.outlook.com>
Message-ID: <49CA0350-E111-4C04-8D5F-D1D7E3E61C9A@anl.gov>


   complex numbers cannot be larger than other complex numbers or smaller. If you want to compare the real parts you can use

     PetscRealPart(a) < PetscRealPart(b)

   similarly for imaginary parts

   Barry
> On Jan 13, 2019, at 5:06 PM, Choudhary, Devyani D via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi,
> 
> I am trying to compare two PetscScalar values, but am getting this error:
> 
> error: no operator "<" matches these operands
> operand types are: PetscScalar < PetscScalar
> 
> Please do share if you know how to resolve this error.
> 
> Sincerely,
> Devyani


From balay at mcs.anl.gov  Sun Jan 13 20:28:42 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Mon, 14 Jan 2019 02:28:42 +0000
Subject: [petsc-users] C++ compilation error
In-Reply-To: <BL0PR07MB52527CCD5DAA2A8C34478ABAD4870@BL0PR07MB5252.namprd07.prod.outlook.com>
References: <BL0PR07MB5252A19A677A64E166B69385D4870@BL0PR07MB5252.namprd07.prod.outlook.com>,
	<alpine.LFD.2.21.1901131402190.10937@sb>,
	<BL0PR07MB52524BE1BBB050F5E2450AB0D4870@BL0PR07MB5252.namprd07.prod.outlook.com>,
	<BN7PR09MB262877C88370E3424C707B64ED870@BN7PR09MB2628.namprd09.prod.outlook.com>
	<BL0PR07MB52527CCD5DAA2A8C34478ABAD4870@BL0PR07MB5252.namprd07.prod.outlook.com>
Message-ID: <alpine.LFD.2.21.1901132027400.10937@sb>

Glad you figured this out. Thanks for the update.

Satish

On Sun, 13 Jan 2019, Choudhary, Devyani D wrote:

> It was the issue with the compiler, and is resolved now. I can run the examples now.
> 
> Thank you for your help.
> 
> 
> Devyani
> 
> ________________________________
> From: Balay, Satish <balay at mcs.anl.gov>
> Sent: Sunday, January 13, 2019 5:18:22 PM
> To: Choudhary, Devyani D; petsc-users
> Subject: RE: [petsc-users] C++ compilation error
> 
> 
> Do you get this error with any petsc example ? say src/ksp/ksp/examples /tutorials/ex2 ?
> 
> 
> 
> Also, copy paste the complete output from make ? when building example or your Test code.
> 
> 
> 
> Satish
> 
> 
> 
> ________________________________
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Choudhary, Devyani D via petsc-users <petsc-users at mcs.anl.gov>
> Sent: Sunday, January 13, 2019 2:27:05 PM
> To: petsc-users
> Subject: Re: [petsc-users] C++ compilation error
> 
> 
> Thank you for the response Satish, but I am still encountering the same issue.
> 
> Following is my makefile:
> 
> 
> PETSC_DIR=/usr/local/pacerepov1/petsc/3.8.3/mvapich2-2.1/intel-15.0/opt/lib/petsc/
> 
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
> include ${PETSC_DIR}/conf/test
> 
> hello: hello.o  chkopts
>         -${CLINKER} -o hello hello.o  ${PETSC_LIB}
>         ${RM} hello.o
> 
> It is in the format of petsc examples. I am not sure why I am still getting the error.
> 
> Devyani
> 
> 
> 
> ________________________________
> From: Balay, Satish <balay at mcs.anl.gov>
> Sent: Sunday, January 13, 2019 3:07:58 PM
> To: Choudhary, Devyani D
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] C++ compilation error
> 
> Presumably PETSc is buit with intel compilers - but somehow gcc/g++ is
> getting used via your makefile.
> 
> Do you get this error when you use PETSc example with a petsc
> makefile? Perhaps you need to format your makefile using petsc
> makefile format. For ex: check
> src/tao/leastsquares/examples/tutorials/makefile
> 
> Satish
> 
> On Sun, 13 Jan 2019, Choudhary, Devyani D via petsc-users wrote:
> 
> > Hi,
> >
> >
> > I am trying to make a simple hello world script using a makefile that includes petsc, and am getting the error
> >
> > "g++: error: unrecognized command line option ?-wd1572?"
> >
> > I am not sure how to fix it.
> >
> > Please let me know if you have some ideas on how to fix this.
> >
> >
> > Thank you
> >
> 

From knepley at gmail.com  Mon Jan 14 00:58:17 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 14 Jan 2019 01:58:17 -0500
Subject: [petsc-users] Example for adaptive high-order FEM in PETSc
In-Reply-To: <CAMYG4G=ETOyWiVJ-sABZypZs74D42NO4My=DXmf6V4H0KtWZPw@mail.gmail.com>
References: <8000A760-4A57-44BF-B698-D43C2C05E1F8@gmail.com>
	<503BFF29-2A57-4CEA-ABFC-C12C711FF76C@gmail.com>
	<0442919E-F0BF-463A-B12D-E67054A94B11@gmail.com>
	<CAMYG4G=ETOyWiVJ-sABZypZs74D42NO4My=DXmf6V4H0KtWZPw@mail.gmail.com>
Message-ID: <CAMYG4GkLTDAG0imtyebTT-Yp1reYXHixmWc727PCWzNSXR9B5Q@mail.gmail.com>

On Sun, Jan 13, 2019 at 3:01 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Jan 12, 2019 at 6:59 PM Manav Bhatia via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I have been studying the source code and the manual entries in the
>> documentation and have a better understanding of setting up a high-order DG
>> analysis.
>>
>> I have been able to identify the routines that set the order of the space
>> and its continuity (CG/DG).
>>
>> The following is still unclear and I would appreciate some guidance:
>>
>> ? A implicit DG solve of the inviscid Euler equations would require
>> Jacobian contributions from the Riemann Solver at element interfaces. I do
>> not see the ability to define Jacobian contribution in
>> PetscDSSetRiemannSolver. What would be the best approach to do this?
>>
>
> You would set a BdResidual/BdJacobian function. You need my branch that
> makes these integrals 2-sided. However, you would also probably need to
> make an aux field with the geometric quantities you need. I can help you do
> that. Then we could roll that up and put it in the library.
>
>
>> ? It appears that the only two choices for polynomials are Lagrange and
>> ?simple?. I am not clear what the ?simple? basis implies. Are these just
>> powers of x, y and z?
>>
>
> Correct. We would have to stick in another polynomial space. This is
> actually pretty easy. We already have code for Legendre in the
> PetscQuadrature code, and we should also probably add Chebyshev.
>

I was not saying the right thing here. We already use Legendre for
polynomials spaces. This is fine until very high order when you need to use
Dubiner to maintain accuracy.

   Matt


> In addition, you probably want GLL quadrature. Some frustrating
> people put GLL code in TS, which should be pulled out and made a
> PetscQuadrature. You can see that this is some coding to do, but
> none of it is design/hard, it just needs to be done.
>
>   Thanks,
>
>     Matt
>
>
>> ? Are Legendre polynomials available elsewhere in PETSc? I saw some
>> mention of it in TS, but not in FE.
>>
>> Thanks,
>> Manav
>>
>>
>>
>> > On Jan 12, 2019, at 2:46 AM, Manav Bhatia <bhatiamanav at gmail.com>
>> wrote:
>> >
>> > While looking through some examples I see that both PetscDSSetJacobian
>> and DMPlexSNESComputeJacobianFEM are being used to set the functions to
>> evaluate the Jacobian contributions on each rank.
>> >
>> > This is my understanding after studying these (please correct if I am
>> wrong):
>> >
>> > ? The PetscDSSetJacobian (and its corresponding residual routine) does
>> not seem to have the concept of an element. Instead, it provides the
>> derivatives of solution at a point and requires that the coefficients for
>> each form g_0, g_1, ? be evaluated. Then, PETSc combines that in the
>> background.
>> >
>> > ? DMPlexSNESComputeJacobianFEM is a method that calls
>> PetscDSSetJacobian to compute the Jacobian.
>> >
>> >
>> > If this is correct, I am a bit confused about the following:
>> >
>> > ? For a DG solve, I can see that PetscDSSetRiemannSolver can be used to
>> compute the convective flux at the interface of two elements. However, I
>> don?t see how the Jacobian contribution of this flux can be computed and
>> added to the system Jacobian.
>> >
>> > ? A more conventional FEM assembly iterates over elements, then over
>> the quadrature points, where the shape functions and their derivatives are
>> initialized and used for computation of residual and Jacobian. Is it
>> possible to follow this procedure in PETSc with DMPlex/DMForest and still
>> use multigrid? Is there an example that demonstrates this?
>> >
>> > ? Is there some form of an index for elements that exist on the local
>> processor that can be used to iterate on local elements?
>> >
>> >
>> > I have tried to look, but could not find a document that outlines these
>> concepts. Is there one that exists and I have missed it?
>> >
>> > I would greatly appreciate some direction.
>> >
>> > Regards,
>> > Manav
>> >
>> >
>> >
>> >
>> >> On Jan 11, 2019, at 10:03 PM, Manav Bhatia <bhatiamanav at gmail.com>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >>  I am interested in setting up a high-order DG FEM solution with PETSc
>> on an octree mesh with adaptivity and geometric multigrid preconditioning.
>> Is there an example that demonstrates this with either DMForest or DMPlex.
>> >>
>> >> Thanks,
>> >> Manav
>> >>
>> >>
>> >
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/cc1bf668/attachment-0001.html>

From niko.karin at gmail.com  Mon Jan 14 02:13:46 2019
From: niko.karin at gmail.com (Karin&NiKo)
Date: Mon, 14 Jan 2019 09:13:46 +0100
Subject: [petsc-users] Using real and complex matrices together
In-Reply-To: <f760a47783c9424e99b9cc2135222a52@wppexc05.purdue.lcl>
References: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>
	<87pnt0fnzp.fsf@jedbrown.org>
	<f760a47783c9424e99b9cc2135222a52@wppexc05.purdue.lcl>
Message-ID: <CA+gX-L-+RicuN1K8w_teEwZhz0Ta+pKtJ7Z-gTp8HgRMNvmfPQ@mail.gmail.com>

Dear Michael,

I  also have plans to use PETSc on real and complex objects and, at the
moment, I do not see how to do it.
If possible, could you please comment on how you defined your wrappers and
how you link the symbols with the adequate functions? Since both PETSc real
and complex  symbols have the same names, I wonder if you use the paths to
the different symbols? Also, did you write your wrappers by hand and did
you write a script to do it (the hidden question is : do you cover all the
objects and methods of the library)?

I thank you in advance,
Nicolas

Le dim. 13 janv. 2019 ? 22:13, Povolotskyi, Mykhailo via petsc-users <
petsc-users at mcs.anl.gov> a ?crit :

> Hi Evan,
>
> I?m one of those users, who has multiple shared libraries.
>
> It is quite stable since I create wrappers around the PETSc functions and
> hide the native PETSc symbols.
>
> But, yes, the interoperations are not possible.
>
> Michael.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Jed Brown via petsc-users <petsc-users at mcs.anl.gov>
> *Sent: *Sunday, January 13, 2019 12:48 PM
> *To: *Evan Um <evanum at gmail.com>; petsc-users <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] Using real and complex matrices together
>
>
> It is not supported, sorry.  Some users do it with careful linking via
> multiple shared libraries, but that is fragile and doesn't interoperate.
>
> Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Hi PETSC users,
> >
> > Inside my codes, I have both complex and real matrices. To solve complex
> > matrices, I build and use complex version of PETSC. In this case, is
> there
> > any way to use real matrices too? Do you have such a PETSC code example?
> > Until now, I have used complex matrices to represent real matrices by
> > setting imaginary components of the real matrices to zero. Real value
> > vectors are also realized by setting imaginary components to zero.
> >
> > In advance, thanks for your comments.
> >
> > Evan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/ca9a2a15/attachment.html>

From tempohoper at gmail.com  Mon Jan 14 07:05:04 2019
From: tempohoper at gmail.com (Sal Am)
Date: Mon, 14 Jan 2019 13:05:04 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
Message-ID: <CAE3k7C6PX4x29g+Hmu2ihKrn5LaS3_H+LTyJEMpKyj1OfBO1eg@mail.gmail.com>

I ran it by: mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
-pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
The error:

[6]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[6]PETSC ERROR: Out of memory. This could be due to allocating
[6]PETSC ERROR: too large an object or bleeding by not properly
[6]PETSC ERROR: destroying unneeded objects.
[6]PETSC ERROR: Memory allocated 0 Memory used by process 39398023168
[6]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[6]PETSC ERROR: Memory requested 18446744066024411136
[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002
Mon Jan 14 08:54:45 2019
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
--with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
--with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
--with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
--download-parmetis --download-metis --download-ptscotch
--download-superlu_dist --download-mumps --with-scalar-type=complex
--with-debugging=yes --download-scalapack --download-superlu
--download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #2 PetscMallocA() line 397 in
/lustre/home/vef002/petsc/src/sys/memory/mal.c
[6]PETSC ERROR: #3 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #4 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 1203 in
/lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #5 MatTransposeMatMult() line 9984 in
/lustre/home/vef002/petsc/src/mat/interface/matrix.c
[6]PETSC ERROR: #6 PCGAMGCoarsen_AGG() line 882 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[6]PETSC ERROR: #7 PCSetUp_GAMG() line 522 in
/lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[6]PETSC ERROR: #8 PCSetUp() line 932 in
/lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: #9 KSPSetUp() line 391 in
/lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: #10 main() line 68 in
/home/vef002/debugenv/tests/solveCmplxLinearSys.cpp
[6]PETSC ERROR: PETSc Option Table entries:
[6]PETSC ERROR: -ksp_monitor
[6]PETSC ERROR: -ksp_type bcgs
[6]PETSC ERROR: -log_view
[6]PETSC ERROR: -malloc off
[6]PETSC ERROR: -mattransposematmult_via scalable
[6]PETSC ERROR: -pc_type gamg
[6]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD
with errorcode 55.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Memory requested error seems astronomical though.... This was done on a
machine with 500GB of memory, during my last check it was using 30GB
mem/processor not sure if it increased suddenly. The file size of the
matrix is 40GB still same matrix
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279



On Fri, Jan 11, 2019 at 5:34 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Add option '-mattransposematmult_via scalable'
> Hong
>
> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I saw the following error message in your first email.
>>
>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>> [0]PETSC ERROR: too large an object or bleeding by not properly
>> [0]PETSC ERROR: destroying unneeded objects.
>>
>> Probably the matrix is too large. You can try with more compute nodes,
>> for example, use 8 nodes instead of 2, and see what happens.
>>
>> --Junchao Zhang
>>
>>
>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Using a larger problem set with 2B non-zero elements and a matrix of 25M
>>> x 25M I get the following error:
>>> [4]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [4]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [4]PETSC ERROR: likely location of problem given in stack below
>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [4]PETSC ERROR:       is given.
>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>> [4]PETSC ERROR: [4]
>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>> [4]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [4]PETSC ERROR: Signal received
>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>> vef002 Fri Jan 11 09:13:23 2019
>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>> --download-parmetis --download-metis --download-ptscotch
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=yes --download-scalapack --download-superlu
>>> --download-fblaslapack=1 --download-cmake
>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>>
>>> --------------------------------------------------------------------------
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>
>>> Using Valgrind on only one of the valgrind files the following error was
>>> written:
>>>
>>> ==9053== Invalid read of size 4
>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>> (matmatmult.c:790)
>>> ==9053==    by 0x5D106F8:
>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>> (mpimatmatmult.c:1186)
>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>> free'd
>>> ==9053==
>>>
>>>
>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>
>>>> Thank you Dave,
>>>>
>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the code
>>>> again with the following options:
>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>> gamg -log_view
>>>> (as on the petsc website you linked)
>>>>
>>>> It finished solving using the iterative solver, but the resulting
>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>
>>>>
>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> I am not sure what is exactly is wrong as the error changes slightly
>>>>>> every time I run it (without changing the parameters).
>>>>>>
>>>>>
>>>>> This likely implies that you have a memory error in your code (a
>>>>> memory leak would not cause this behaviour).
>>>>> I strongly suggest you make sure your code is free of memory errors.
>>>>> You can do this using valgrind. See here
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>
>>>>> for an explanation of how to use valgrind.
>>>>>
>>>>>
>>>>>> I have attached the first two run's errors and my code.
>>>>>>
>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>
>>>>>> how I run it in the job script on 2 node with 32 processors using the
>>>>>> clusters OpenMPI.
>>>>>>
>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>
>>>>>> the matrix:
>>>>>> 2 122 821 366 (non-zero elements)
>>>>>> 25 947 279 x 25 947 279
>>>>>>
>>>>>> Thanks and all the best
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/dfa7faae/attachment-0001.html>

From knepley at gmail.com  Mon Jan 14 07:26:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 14 Jan 2019 08:26:16 -0500
Subject: [petsc-users] PETSC for singular system
In-Reply-To: <DM5PR22MB02654E74068896C2159784EAA8800@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265550095AF778E3080FAC4A8870@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gnv9-S6nhPZn60t=8B-ASRyROJqVqfQqKSeLe=bsL+10w@mail.gmail.com>
	<DM5PR22MB02654E74068896C2159784EAA8800@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4G=PqSfjrBivoXb7PKr1aynsJ2AxQHM7uf=eLFt6vn6eBg@mail.gmail.com>

On Mon, Jan 14, 2019 at 7:55 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> So must I  figure out the index of the zero columns and row to get the
> null space first, And then I remove it to generator Cholesky or LU
> preconditionor?Is this case, should the nontrivial null space be (1,0,0,0)?
>

No

1) If you have a consistent rhs, then CG works as is. However, you should
put it in in case your preconditioner introduces nullspace components.

2) Factorization will NEVER work on singular systems, unless you use SVD,
which is only for very small systems.

  Thanks,

    Matt


> Thanks
> Get Outlook for iOS <https://aka.ms/o0ukef>
> _____________________________
> From: Matthew Knepley <knepley at gmail.com>
> Sent: Monday, January 14, 2019 2:00 AM
> Subject: Re: [petsc-users] PETSC for singular system
> To: Yaxiong Chen <chen2018 at purdue.edu>, PETSc <petsc-maint at mcs.anl.gov>
>
>
> On Sun, Jan 13, 2019 at 6:59 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
>> Hello Matthew,
>>
>>
>> Does this mean in PETSC , CG is not suitable for semi positive definite
>> system?
>>
>
> No, it means factorization is not. CG is fine.
>
>   Matt
>
>
>>  My  system is equivalent to a SPD system with arbitrary extra rows and
>> columns of zeros. I expect the same iteration process will lead to xT=(0,1,1,1)
>> with initial guess 0, since the rows of 0 will not influence the
>> corresponding solution. Is this possible to be realized with CG in PETSC?
>>
>>
>> Thanks
>>
>>
>> Yaxiong Chen,
>>
>> Ph.D. Student
>>
>> School of Mechanical Engineering, 3171
>>
>> 585 Purdue Mall
>>
>> West Lafayette, IN 47907
>>
>>
>>
>>
>>
>> ------------------------------
>> *From:* Matthew Knepley <knepley at gmail.com>
>> *Sent:* Friday, January 11, 2019 2:30 PM
>> *To:* Yaxiong Chen
>> *Cc:* PETSc
>> *Subject:* Re: [petsc-users] PETSC for singular system
>>
>> On Fri, Jan 11, 2019 at 10:04 AM Yaxiong Chen via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>> Hello,
>>
>>
>> I am trying to use PETSC to solve a singular system like the following.
>>
>> .
>>
>> This equals inserting rows and columns at arbitrary place(Actually I
>> even don't know whether it is singular or not. Sometimes some degree of
>> freedom x_i may not participate in the global matrix)
>>
>>
>> I searched some discussion online and tried to remove the nullspace with
>> the following code:
>>
>>      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>>      call KSPSetOperators(ksp,Amat,Amat,ierr)
>>      call KSPSetFromOptions(ksp,ierr)
>>      call KSPSetType(ksp, KSPCG,ierr)
>>      call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, dummyVecs,
>> nullspace, ierr)
>>      call MatSetNullSpace(Amat,nullspace,ierr)
>>      call MatNullSpaceDestroy(nullspace,ierr)
>>      call KSPSolve(ksp,bvec,xvec,ierr)
>>
>> But it still gives me some error saying Zero pivot in LU factorization.
>> I am just wondering how MatSetNullSpace() can handle this?
>>
>>
>> If you give the nullspace, the Kryloc method will work. However, you
>> still cannot use factorization. So
>>
>> 1) -ksp_type gmres -pc_type jacobi will solve your system
>>
>> 2) -ksp_type gmres -pc_type svd will solve your system
>>
>> Saying anything more general than that is really not possible.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>> Thanks
>>
>> Yaxiong Chen,
>>
>> School of Mechanical Engineering, 3171
>>
>> 585 Purdue Mall
>>
>> West Lafayette, IN 47907
>>
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/f2ffcee9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 2573 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/f2ffcee9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3538 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/f2ffcee9/attachment-0001.png>

From stefano.zampini at gmail.com  Mon Jan 14 07:31:04 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 14 Jan 2019 16:31:04 +0300
Subject: [petsc-users] PETSC for singular system
In-Reply-To: <CAMYG4G=PqSfjrBivoXb7PKr1aynsJ2AxQHM7uf=eLFt6vn6eBg@mail.gmail.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265550095AF778E3080FAC4A8870@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gnv9-S6nhPZn60t=8B-ASRyROJqVqfQqKSeLe=bsL+10w@mail.gmail.com>
	<DM5PR22MB02654E74068896C2159784EAA8800@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=PqSfjrBivoXb7PKr1aynsJ2AxQHM7uf=eLFt6vn6eBg@mail.gmail.com>
Message-ID: <CAGPUishPwf2LT3mTib-wQr75POUoEBWfqkD9UiE+jdmSfUi-PA@mail.gmail.com>

If you want to use factorizations, then you have to make sure that you
preallocate at least one entry (the diagonal one) for each empty row, and
put a one there. The solution will be equivalent (modulo nullspace)

Il giorno lun 14 gen 2019 alle ore 16:27 Matthew Knepley via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

> On Mon, Jan 14, 2019 at 7:55 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
>> So must I  figure out the index of the zero columns and row to get the
>> null space first, And then I remove it to generator Cholesky or LU
>> preconditionor?Is this case, should the nontrivial null space be (1,0,0,0)?
>>
>
> No
>
> 1) If you have a consistent rhs, then CG works as is. However, you should
> put it in in case your preconditioner introduces nullspace components.
>
> 2) Factorization will NEVER work on singular systems, unless you use SVD,
> which is only for very small systems.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks
>> Get Outlook for iOS <https://aka.ms/o0ukef>
>> _____________________________
>> From: Matthew Knepley <knepley at gmail.com>
>> Sent: Monday, January 14, 2019 2:00 AM
>> Subject: Re: [petsc-users] PETSC for singular system
>> To: Yaxiong Chen <chen2018 at purdue.edu>, PETSc <petsc-maint at mcs.anl.gov>
>>
>>
>> On Sun, Jan 13, 2019 at 6:59 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>>
>>> Hello Matthew,
>>>
>>>
>>> Does this mean in PETSC , CG is not suitable for semi positive definite
>>> system?
>>>
>>
>> No, it means factorization is not. CG is fine.
>>
>>   Matt
>>
>>
>>>  My  system is equivalent to a SPD system with arbitrary extra rows and
>>> columns of zeros. I expect the same iteration process will lead to xT=(0,1,1,1)
>>> with initial guess 0, since the rows of 0 will not influence the
>>> corresponding solution. Is this possible to be realized with CG in PETSC?
>>>
>>>
>>> Thanks
>>>
>>>
>>> Yaxiong Chen,
>>>
>>> Ph.D. Student
>>>
>>> School of Mechanical Engineering, 3171
>>>
>>> 585 Purdue Mall
>>>
>>> West Lafayette, IN 47907
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* Matthew Knepley <knepley at gmail.com>
>>> *Sent:* Friday, January 11, 2019 2:30 PM
>>> *To:* Yaxiong Chen
>>> *Cc:* PETSc
>>> *Subject:* Re: [petsc-users] PETSC for singular system
>>>
>>> On Fri, Jan 11, 2019 at 10:04 AM Yaxiong Chen via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>> Hello,
>>>
>>>
>>> I am trying to use PETSC to solve a singular system like the following.
>>>
>>> .
>>>
>>> This equals inserting rows and columns at arbitrary place(Actually I
>>> even don't know whether it is singular or not. Sometimes some degree of
>>> freedom x_i may not participate in the global matrix)
>>>
>>>
>>> I searched some discussion online and tried to remove the nullspace with
>>> the following code:
>>>
>>>      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>>>      call KSPSetOperators(ksp,Amat,Amat,ierr)
>>>      call KSPSetFromOptions(ksp,ierr)
>>>      call KSPSetType(ksp, KSPCG,ierr)
>>>      call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0,
>>> dummyVecs, nullspace, ierr)
>>>      call MatSetNullSpace(Amat,nullspace,ierr)
>>>      call MatNullSpaceDestroy(nullspace,ierr)
>>>      call KSPSolve(ksp,bvec,xvec,ierr)
>>>
>>> But it still gives me some error saying Zero pivot in LU factorization.
>>> I am just wondering how MatSetNullSpace() can handle this?
>>>
>>>
>>> If you give the nullspace, the Kryloc method will work. However, you
>>> still cannot use factorization. So
>>>
>>> 1) -ksp_type gmres -pc_type jacobi will solve your system
>>>
>>> 2) -ksp_type gmres -pc_type svd will solve your system
>>>
>>> Saying anything more general than that is really not possible.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>> Thanks
>>>
>>> Yaxiong Chen,
>>>
>>> School of Mechanical Engineering, 3171
>>>
>>> 585 Purdue Mall
>>>
>>> West Lafayette, IN 47907
>>>
>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/65a0781e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 2573 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/65a0781e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3538 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/65a0781e/attachment-0003.png>

From mfadams at lbl.gov  Mon Jan 14 07:41:05 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 14 Jan 2019 08:41:05 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C6PX4x29g+Hmu2ihKrn5LaS3_H+LTyJEMpKyj1OfBO1eg@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAE3k7C6PX4x29g+Hmu2ihKrn5LaS3_H+LTyJEMpKyj1OfBO1eg@mail.gmail.com>
Message-ID: <CADOhEh56huxBs9jhe5dGVybojoLn95Cq-yWB1z2d0x=6KeSKQw@mail.gmail.com>

The memory requested is an insane number. You may need to use 64 bit
integers.

On Mon, Jan 14, 2019 at 8:06 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I ran it by: mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
> The error:
>
> [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [6]PETSC ERROR: Out of memory. This could be due to allocating
> [6]PETSC ERROR: too large an object or bleeding by not properly
> [6]PETSC ERROR: destroying unneeded objects.
> [6]PETSC ERROR: Memory allocated 0 Memory used by process 39398023168
> [6]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [6]PETSC ERROR: Memory requested 18446744066024411136
> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [6]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
> vef002 Mon Jan 14 08:54:45 2019
> [6]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
> --download-parmetis --download-metis --download-ptscotch
> --download-superlu_dist --download-mumps --with-scalar-type=complex
> --with-debugging=yes --download-scalapack --download-superlu
> --download-fblaslapack=1 --download-cmake
> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989
> in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> [6]PETSC ERROR: #2 PetscMallocA() line 397 in
> /lustre/home/vef002/petsc/src/sys/memory/mal.c
> [6]PETSC ERROR: #3 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989
> in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> [6]PETSC ERROR: #4 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 1203 in
> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> [6]PETSC ERROR: #5 MatTransposeMatMult() line 9984 in
> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [6]PETSC ERROR: #6 PCGAMGCoarsen_AGG() line 882 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
> [6]PETSC ERROR: #7 PCSetUp_GAMG() line 522 in
> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
> [6]PETSC ERROR: #8 PCSetUp() line 932 in
> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [6]PETSC ERROR: #9 KSPSetUp() line 391 in
> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: #10 main() line 68 in
> /home/vef002/debugenv/tests/solveCmplxLinearSys.cpp
> [6]PETSC ERROR: PETSc Option Table entries:
> [6]PETSC ERROR: -ksp_monitor
> [6]PETSC ERROR: -ksp_type bcgs
> [6]PETSC ERROR: -log_view
> [6]PETSC ERROR: -malloc off
> [6]PETSC ERROR: -mattransposematmult_via scalable
> [6]PETSC ERROR: -pc_type gamg
> [6]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD
> with errorcode 55.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
>
> Memory requested error seems astronomical though.... This was done on a
> machine with 500GB of memory, during my last check it was using 30GB
> mem/processor not sure if it increased suddenly. The file size of the
> matrix is 40GB still same matrix
> 2 122 821 366 (non-zero elements)
> 25 947 279 x 25 947 279
>
>
>
> On Fri, Jan 11, 2019 at 5:34 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Add option '-mattransposematmult_via scalable'
>> Hong
>>
>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> I saw the following error message in your first email.
>>>
>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>> [0]PETSC ERROR: destroying unneeded objects.
>>>
>>> Probably the matrix is too large. You can try with more compute nodes,
>>> for example, use 8 nodes instead of 2, and see what happens.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Using a larger problem set with 2B non-zero elements and a matrix of
>>>> 25M x 25M I get the following error:
>>>> [4]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>>> probably memory access out of range
>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> [4]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>>> OS X to find memory corruption errors
>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>> ------------------------------------
>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>> available,
>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>> function
>>>> [4]PETSC ERROR:       is given.
>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>> [4]PETSC ERROR: [4]
>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>> [4]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [4]PETSC ERROR: Signal received
>>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>>> vef002 Fri Jan 11 09:13:23 2019
>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>> --download-parmetis --download-metis --download-ptscotch
>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>> --download-fblaslapack=1 --download-cmake
>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>
>>>> --------------------------------------------------------------------------
>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>> with errorcode 59.
>>>>
>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>> You may or may not see output from other processes, depending on
>>>> exactly when Open MPI kills them.
>>>>
>>>> --------------------------------------------------------------------------
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>>> batch system) has told this process to end
>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> [0]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>
>>>> Using Valgrind on only one of the valgrind files the following error
>>>> was written:
>>>>
>>>> ==9053== Invalid read of size 4
>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>> (matmatmult.c:790)
>>>> ==9053==    by 0x5D106F8:
>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>> (mpimatmatmult.c:1186)
>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>>> free'd
>>>> ==9053==
>>>>
>>>>
>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>
>>>>> Thank you Dave,
>>>>>
>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the code
>>>>> again with the following options:
>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>> gamg -log_view
>>>>> (as on the petsc website you linked)
>>>>>
>>>>> It finished solving using the iterative solver, but the resulting
>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>
>>>>>
>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> I am not sure what is exactly is wrong as the error changes slightly
>>>>>>> every time I run it (without changing the parameters).
>>>>>>>
>>>>>>
>>>>>> This likely implies that you have a memory error in your code (a
>>>>>> memory leak would not cause this behaviour).
>>>>>> I strongly suggest you make sure your code is free of memory errors.
>>>>>> You can do this using valgrind. See here
>>>>>>
>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>
>>>>>> for an explanation of how to use valgrind.
>>>>>>
>>>>>>
>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>
>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>
>>>>>>> how I run it in the job script on 2 node with 32 processors using
>>>>>>> the clusters OpenMPI.
>>>>>>>
>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>
>>>>>>> the matrix:
>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>> 25 947 279 x 25 947 279
>>>>>>>
>>>>>>> Thanks and all the best
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/a29d0b6e/attachment.html>

From valerio.barnabei at gmail.com  Mon Jan 14 08:08:08 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Mon, 14 Jan 2019 15:08:08 +0100
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
Message-ID: <CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>

Thank you Stefano for your quick answer, I will try to change PC as you
suggest, and I will read something about MATIS type. I will let you know if
I still have doubts.

Valerio

Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <
stefano.zampini at gmail.com> ha scritto:

> What problem are you trying to solve?
> For standard elliptic problems (e.g. poisson or elasticity) you should use
> a preconditioner that scales, for example PCGAMG or BoomerAMG from HYPRE
> (PCHYPRE, if you configured PETSc with support for HYPRE via
> --download-hypre)
> Alternatively, since you are using FEM with local Assembly, you can use
> PCBDDC, but this requires few extra calls
>
> - call MatSetType(K,MATIS)
> - provide an ISLocalToGlobalMapping of local to global degrees of freedom
> before you set preallocation via MatSetLocalToGlobalMapping (if you provide
> this call, then you can use MatSetValuesLocal for both AIJ amd MATIS types)
>
>
>
> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via petsc-users <
> petsc-users at mcs.anl.gov> ha scritto:
>
>> Hello everybody,
>>
>> I have some doubts about the parallel solution of a simple FEM ksp
>> problem. I searched in the mailing list but the information I found are not
>> helping me.
>> I need to control manually the parallel assemble of the main matrix, but
>> I would like to use PetSC for the solution of the final linear system
>> K*U=F.
>> The algorithm I am using, loads an already decomposed FEM domain, and
>> each process contributes to the global matrix assembly using its portion of
>> domain decomposition (made in pre-processing). In the element loop, at each
>> element I load the element submatrix in the global matrix via MatSetValue
>> function. Then, out of the element loop, I write
>> /*
>> ...matrix preallocation
>> ...calculate element local stiffness matrix (local in a FEM sense and
>> local in a domain decomposition sense)
>> MatSetValue (ADD)
>> */
>> and then
>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>>
>> This part scales perfectly and is sufficiently fast.
>> My K matrix is a sort of banded diagonal matrix: my preallocation is not
>> perfect, but it works at this stage of my work.
>> Finally I solve the system with the ksp logic:
>>
>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI, the
>> behaviour is almost identical, the iteration number lowers but is still
>> really high
>>
>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>> ierr=KSPSolve(ksp,F,sol);
>>
>> When I run the code I am noticing some strange behaviours. Indeed, the
>> numerical solution appear right, and the code scales if I make a simple
>> strong-scaling test. However, the ksp_log and log_view (see below, logs for
>> a 160k quad linear element uniform 2dmesh) shows me a huge number of
>> iterations, and the time spent in "solve" function is very high.
>> Furthermore, the more I increase the problem size keeping a constant load
>> per process (weak scaling) the more iterations seems to be needed for
>> convergence. In general, when increasing the problem size (i.e. 16M
>> elements) the convergence is not achieved before the maximum iteration
>> number is achieved.
>>
>> Question 1: Am I doing it right? Or am I making some trouble with the MPI
>> communication management? (i'm aware that my domain decomposition does not
>> match the petsc matrix decomposition, I'm expecting it to cripple my
>> algorithm a bit)
>> Question 2: Why the solver takes so much time and so many iterations? the
>> problem is really simple, and the solution appears correct when plotted. Am
>> I ill conditioning the system?
>> Thanks in advance for your help. I can add any further information if
>> needed.
>>
>> Valerio
>>
>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/7cbbaf96/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Jan 14 09:31:33 2019
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 14 Jan 2019 09:31:33 -0600
Subject: [petsc-users] Using real and complex matrices together
In-Reply-To: <CA+gX-L-+RicuN1K8w_teEwZhz0Ta+pKtJ7Z-gTp8HgRMNvmfPQ@mail.gmail.com>
References: <CAP1yThQ7v0OdgFCcSoYU-u=7C=e75uQ=sT74-mG-ySVdHgCkqw@mail.gmail.com>
	<87pnt0fnzp.fsf@jedbrown.org>
	<f760a47783c9424e99b9cc2135222a52@wppexc05.purdue.lcl>
	<CA+gX-L-+RicuN1K8w_teEwZhz0Ta+pKtJ7Z-gTp8HgRMNvmfPQ@mail.gmail.com>
Message-ID: <A1284B4E-F717-4156-8196-811476708D35@gmail.com>

I have successfully used Petsc built with real numbers to solve system of equations in complex numbers by representing the NxN complex system of equations as 2Nx2N real system of equations. 

A relevant paper is this: 

@Article{Day2001,
author = {Day, David and Heroux, Michael A}, 
title = {Solving complex-valued linear systems via equivalent real formulations}, 
journal = {SIAM Journal on Scientific Computing}, 
volume = {23}, 
number = {2}, 
pages = {480--498}, 
year = {2001}, 
abstract = {Most preconditioned iterative methods apply to both real-and complex-valued linear systems. At the same time, most iterative linear solver packages available today focus exclusively on real-valued systems or deal with complex-valued systems as an.}, 
location = {}, 
keywords = {}}

Note that it is important to solve this as block matrix, which is the point made in the paper. 

My implementation in MAST is here: https://github.com/MASTmultiphysics/mast-multiphysics/blob/7b5c414e3d85a2712b3054d10be9cddb1b288a82/src/solver/complex_solver_base.cpp#L333


-Manav

> On Jan 14, 2019, at 2:13 AM, Karin&NiKo via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear Michael,
> 
> I  also have plans to use PETSc on real and complex objects and, at the moment, I do not see how to do it.
> If possible, could you please comment on how you defined your wrappers and how you link the symbols with the adequate functions? Since both PETSc real and complex  symbols have the same names, I wonder if you use the paths to the different symbols? Also, did you write your wrappers by hand and did you write a script to do it (the hidden question is : do you cover all the objects and methods of the library)?
> 
> I thank you in advance,
> Nicolas
> 
> Le dim. 13 janv. 2019 ? 22:13, Povolotskyi, Mykhailo via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> a ?crit :
> Hi Evan,
> 
> I?m one of those users, who has multiple shared libraries.
> 
> It is quite stable since I create wrappers around the PETSc functions and hide the native PETSc symbols.
> 
> But, yes, the interoperations are not possible.
> 
> Michael.
> 
>  
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
> 
>  
> From: Jed Brown via petsc-users <mailto:petsc-users at mcs.anl.gov>
> Sent: Sunday, January 13, 2019 12:48 PM
> To: Evan Um <mailto:evanum at gmail.com>; petsc-users <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Using real and complex matrices together
> 
>  
> It is not supported, sorry.  Some users do it with careful linking via
> multiple shared libraries, but that is fragile and doesn't interoperate.
> 
> Evan Um via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> writes:
> 
> > Hi PETSC users,
> >
> > Inside my codes, I have both complex and real matrices. To solve complex
> > matrices, I build and use complex version of PETSC. In this case, is there
> > any way to use real matrices too? Do you have such a PETSC code example?
> > Until now, I have used complex matrices to represent real matrices by
> > setting imaginary components of the real matrices to zero. Real value
> > vectors are also realized by setting imaginary components to zero.
> >
> > In advance, thanks for your comments.
> >
> > Evan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/55673089/attachment.html>

From hzhang at mcs.anl.gov  Mon Jan 14 10:36:31 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Mon, 14 Jan 2019 16:36:31 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CADOhEh56huxBs9jhe5dGVybojoLn95Cq-yWB1z2d0x=6KeSKQw@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAE3k7C6PX4x29g+Hmu2ihKrn5LaS3_H+LTyJEMpKyj1OfBO1eg@mail.gmail.com>
	<CADOhEh56huxBs9jhe5dGVybojoLn95Cq-yWB1z2d0x=6KeSKQw@mail.gmail.com>
Message-ID: <CAGCphBurDXgAsx9GC3xKFpYF1Ww-AW8nhmCEfPa3wYfA4+JAqw@mail.gmail.com>

This time, it crashes at
[6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
ierr      = PetscMalloc1(bi[pn]+1,&bj);
which allocates local portion of B^T*A.
You may also try to increase number of cores to reduce local matrix size.
Hong


On Mon, Jan 14, 2019 at 7:42 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
The memory requested is an insane number. You may need to use 64 bit integers.

On Mon, Jan 14, 2019 at 8:06 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I ran it by: mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
The error:

[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Out of memory. This could be due to allocating
[6]PETSC ERROR: too large an object or bleeding by not properly
[6]PETSC ERROR: destroying unneeded objects.
[6]PETSC ERROR: Memory allocated 0 Memory used by process 39398023168
[6]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[6]PETSC ERROR: Memory requested 18446744066024411136
[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Mon Jan 14 08:54:45 2019
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #2 PetscMallocA() line 397 in /lustre/home/vef002/petsc/src/sys/memory/mal.c
[6]PETSC ERROR: #3 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #4 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 1203 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[6]PETSC ERROR: #5 MatTransposeMatMult() line 9984 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[6]PETSC ERROR: #6 PCGAMGCoarsen_AGG() line 882 in /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[6]PETSC ERROR: #7 PCSetUp_GAMG() line 522 in /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[6]PETSC ERROR: #8 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: #9 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: #10 main() line 68 in /home/vef002/debugenv/tests/solveCmplxLinearSys.cpp
[6]PETSC ERROR: PETSc Option Table entries:
[6]PETSC ERROR: -ksp_monitor
[6]PETSC ERROR: -ksp_type bcgs
[6]PETSC ERROR: -log_view
[6]PETSC ERROR: -malloc off
[6]PETSC ERROR: -mattransposematmult_via scalable
[6]PETSC ERROR: -pc_type gamg
[6]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD
with errorcode 55.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Memory requested error seems astronomical though.... This was done on a machine with 500GB of memory, during my last check it was using 30GB mem/processor not sure if it increased suddenly. The file size of the matrix is 40GB still same matrix
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279



On Fri, Jan 11, 2019 at 5:34 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Add option '-mattransposematmult_via scalable'
Hong

On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/2a5b5d82/attachment-0001.html>

From fdkong.jd at gmail.com  Mon Jan 14 11:26:59 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Mon, 14 Jan 2019 10:26:59 -0700
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
Message-ID: <CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>

Hi Hong,

According to this PR
https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff

Should we set the scalable algorithm as default?

Thanks,

Fande Kong,

On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Add option '-mattransposematmult_via scalable'
> Hong
>
> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I saw the following error message in your first email.
>>
>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>> [0]PETSC ERROR: too large an object or bleeding by not properly
>> [0]PETSC ERROR: destroying unneeded objects.
>>
>> Probably the matrix is too large. You can try with more compute nodes,
>> for example, use 8 nodes instead of 2, and see what happens.
>>
>> --Junchao Zhang
>>
>>
>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Using a larger problem set with 2B non-zero elements and a matrix of 25M
>>> x 25M I get the following error:
>>> [4]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [4]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [4]PETSC ERROR: likely location of problem given in stack below
>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [4]PETSC ERROR:       is given.
>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>> [4]PETSC ERROR: [4]
>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>> [4]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [4]PETSC ERROR: Signal received
>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>> vef002 Fri Jan 11 09:13:23 2019
>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>> --download-parmetis --download-metis --download-ptscotch
>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>> --with-debugging=yes --download-scalapack --download-superlu
>>> --download-fblaslapack=1 --download-cmake
>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>>
>>> --------------------------------------------------------------------------
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>
>>> Using Valgrind on only one of the valgrind files the following error was
>>> written:
>>>
>>> ==9053== Invalid read of size 4
>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>> (matmatmult.c:790)
>>> ==9053==    by 0x5D106F8:
>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>> (mpimatmatmult.c:1186)
>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>> free'd
>>> ==9053==
>>>
>>>
>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>
>>>> Thank you Dave,
>>>>
>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the code
>>>> again with the following options:
>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>> gamg -log_view
>>>> (as on the petsc website you linked)
>>>>
>>>> It finished solving using the iterative solver, but the resulting
>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>
>>>>
>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> I am not sure what is exactly is wrong as the error changes slightly
>>>>>> every time I run it (without changing the parameters).
>>>>>>
>>>>>
>>>>> This likely implies that you have a memory error in your code (a
>>>>> memory leak would not cause this behaviour).
>>>>> I strongly suggest you make sure your code is free of memory errors.
>>>>> You can do this using valgrind. See here
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>
>>>>> for an explanation of how to use valgrind.
>>>>>
>>>>>
>>>>>> I have attached the first two run's errors and my code.
>>>>>>
>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>
>>>>>> how I run it in the job script on 2 node with 32 processors using the
>>>>>> clusters OpenMPI.
>>>>>>
>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>
>>>>>> the matrix:
>>>>>> 2 122 821 366 (non-zero elements)
>>>>>> 25 947 279 x 25 947 279
>>>>>>
>>>>>> Thanks and all the best
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/4c3be893/attachment.html>

From bsmith at mcs.anl.gov  Mon Jan 14 13:15:05 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 14 Jan 2019 19:15:05 +0000
Subject: [petsc-users] PETSC for singular system
In-Reply-To: <CAMYG4G=PqSfjrBivoXb7PKr1aynsJ2AxQHM7uf=eLFt6vn6eBg@mail.gmail.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265550095AF778E3080FAC4A8870@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gnv9-S6nhPZn60t=8B-ASRyROJqVqfQqKSeLe=bsL+10w@mail.gmail.com>
	<DM5PR22MB02654E74068896C2159784EAA8800@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=PqSfjrBivoXb7PKr1aynsJ2AxQHM7uf=eLFt6vn6eBg@mail.gmail.com>
Message-ID: <8F0FB648-24BB-4A3B-936A-F7BA37DA6F09@anl.gov>



> On Jan 14, 2019, at 7:26 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Mon, Jan 14, 2019 at 7:55 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> So must I  figure out the index of the zero columns and row to get the null space first, And then I remove it to generator Cholesky or LU preconditionor?Is this case, should the nontrivial null space be (1,0,0,0)?
> 
> No
> 
> 1) If you have a consistent rhs, then CG works as is. However, you should put it in in case your preconditioner introduces nullspace components.
> 
> 2) Factorization will NEVER work on singular systems, unless you use SVD, which is only for very small systems.
> 
>   Thanks,
> 
>     Matt
>  
> Thanks
> Get Outlook for iOS
> _____________________________
> From: Matthew Knepley <knepley at gmail.com>
> Sent: Monday, January 14, 2019 2:00 AM
> Subject: Re: [petsc-users] PETSC for singular system
> To: Yaxiong Chen <chen2018 at purdue.edu>, PETSc <petsc-maint at mcs.anl.gov>
> 
> 
> On Sun, Jan 13, 2019 at 6:59 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> Hello Matthew,
> 
> 
> 
> Does this mean in PETSC , CG is not suitable for semi positive definite system?
> 
> No, it means factorization is not. CG is fine.
> 
>   Matt
>  
>  My  system is equivalent to a SPD system with arbitrary extra rows and columns of zeros. I expect the same iteration process will lead to xT=(0,1,1,1) with initial guess 0, since the rows of 0 will not influence the corresponding solution. Is this possible to be realized with CG in PETSC?

    This is only true for unpreconditioned CG or CG with some trivial preconditioner; but these will converge slowly and not be practical.

     What you need to do is go back to your original formulation and construct a matrix that does not have the zero rows and columns; they are not needed and just mess up the solver big time.

    Barry

> 
> Thanks
> 
> 
> Yaxiong Chen, 
> 
> Ph.D. Student 
> 
> 
> School of Mechanical Engineering, 3171 
> 
> 585 Purdue Mall
> 
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> 
> 
> 
> From: Matthew Knepley <knepley at gmail.com>
> Sent: Friday, January 11, 2019 2:30 PM
> To: Yaxiong Chen
> Cc: PETSc
> Subject: Re: [petsc-users] PETSC for singular system
>  
> On Fri, Jan 11, 2019 at 10:04 AM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello, 
> 
> I am trying to use PETSC to solve a singular system like the following.
> <pastedImage.png>.
> This equals inserting rows and columns at arbitrary place(Actually I even don't know whether it is singular or not. Sometimes some degree of freedom x_i may not participate in the global matrix)
> <pastedImage.png>
> I searched some discussion online and tried to remove the nullspace with the following code:
>      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>      call KSPSetOperators(ksp,Amat,Amat,ierr)
>      call KSPSetFromOptions(ksp,ierr)
>      call KSPSetType(ksp, KSPCG,ierr) 
>      call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, dummyVecs, nullspace, ierr)
>      call MatSetNullSpace(Amat,nullspace,ierr)
>      call MatNullSpaceDestroy(nullspace,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
> 
> But it still gives me some error saying Zero pivot in LU factorization.
> I am just wondering how MatSetNullSpace() can handle this?
> 
> If you give the nullspace, the Kryloc method will work. However, you still cannot use factorization. So
> 
> 1) -ksp_type gmres -pc_type jacobi will solve your system
> 
> 2) -ksp_type gmres -pc_type svd will solve your system
> 
> Saying anything more general than that is really not possible.
> 
>   Thanks,
> 
>     Matt
>  
> Thanks
> 
> 
> Yaxiong Chen, 
> 
> School of Mechanical Engineering, 3171 
> 
> 585 Purdue Mall
> 
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From hzhang at mcs.anl.gov  Mon Jan 14 13:29:20 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Mon, 14 Jan 2019 19:29:20 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
Message-ID: <CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>

Fande:
According to this PR https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff

Should we set the scalable algorithm as default?
Sure, we can. But I feel we need do more tests to compare scalable and non-scalable algorithms.
On theory, for small to medium matrices, non-scalable matmatmult() algorithm enables more efficient
data accessing. Andreas optimized scalable implementation. Our non-scalable implementation might have room to be further optimized.
Hong

On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Add option '-mattransposematmult_via scalable'
Hong

On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/b860e823/attachment-0001.html>

From maahi.buet at gmail.com  Mon Jan 14 19:08:44 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Mon, 14 Jan 2019 20:08:44 -0500
Subject: [petsc-users] PETSc turtorial example generating unexpected result
Message-ID: <CAJOaupFo2dMMh9v8EBsUxuGHoX1LXr3824PHpfQ2oO9LOhxpXw@mail.gmail.com>

Hello all,

I complied and run the example *ex2f.F90* located in
*/home/maahi/petsc/src/ksp/ksp/examples/tutorials.
*As in the code the right hand side vector, b, was computed assuming an
exact solution of a vector with all elements of 1.0, all the values of
vector 'x' should have been 1.0. But after running the code as it was
provided, the values of vector x that I get are the followings -
 2.72322e-07
3.81437e-07
1.58922e-07
3.81437e-07
2.38878e-07
-6.65645e-07
1.58922e-07
-6.65645e-07
-2.51219e-07

I am wondering what went wrong? Could you please help me with some insight
into this?

Regards,
Maahi Talukder
Clarkson University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/4e400b3b/attachment.html>

From mfadams at lbl.gov  Mon Jan 14 20:33:23 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 14 Jan 2019 21:33:23 -0500
Subject: [petsc-users] PETSc turtorial example generating unexpected
 result
In-Reply-To: <CAJOaupFo2dMMh9v8EBsUxuGHoX1LXr3824PHpfQ2oO9LOhxpXw@mail.gmail.com>
References: <CAJOaupFo2dMMh9v8EBsUxuGHoX1LXr3824PHpfQ2oO9LOhxpXw@mail.gmail.com>
Message-ID: <CADOhEh4b=R1oUDrBCbwAuUBph9ESdyLuqRR0wOYg9joz=yXdSA@mail.gmail.com>

This code puts the error in x:

call VecAXPY(x,neg_one,u,ierr)

I suspect you printed these numbers after this statement.

On Mon, Jan 14, 2019 at 8:10 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello all,
>
> I complied and run the example *ex2f.F90* located in  */home/maahi/petsc/src/ksp/ksp/examples/tutorials.
> *As in the code the right hand side vector, b, was computed assuming an
> exact solution of a vector with all elements of 1.0, all the values of
> vector 'x' should have been 1.0. But after running the code as it was
> provided, the values of vector x that I get are the followings -
>  2.72322e-07
> 3.81437e-07
> 1.58922e-07
> 3.81437e-07
> 2.38878e-07
> -6.65645e-07
> 1.58922e-07
> -6.65645e-07
> -2.51219e-07
>
> I am wondering what went wrong? Could you please help me with some insight
> into this?
>
> Regards,
> Maahi Talukder
> Clarkson University
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190114/d14b085d/attachment.html>

From leejearl at mail.nwpu.edu.cn  Tue Jan 15 03:51:14 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Tue, 15 Jan 2019 17:51:14 +0800
Subject: [petsc-users] How Can I scatter a global vector to a local vector
 without using DMGlobaltoLocalBegin/DMGlobaltoLocalEnd?
Message-ID: <8579311c1a5e115c00969f06b9ad22888f7b20d1.camel@mail.nwpu.edu.cn>

Hi, Matt

> > Hi, all Petscer:
> >   I am very sorry for disturbing you for my questions.
> 
> 
> I like your questions.
> 
> 
> > I have a
> > problems as follow.
> >   1. A DMPlex object dm is created.
> >   2. A global vector and a local vector are created using the
> following
> > routines DMCreateGlobalVector and DMCreateLocalVector.
> >   3. I have initialed global Vector. Because the local vector have
> some
> > overlap cells, I want to scatter the data from global to local.
> >   4. If I don't use the routines
> > DMGlobaltoLocalBegin/DMGlobaltoLocalEnd, how can I scatter it?
> 
> 
> To understand the question better, why would you not use
> DMGlobalToLocal()?
I have create a PetscSection for the dm object. The section is used to
store the macroscopic variables $\rho$, u,  v, $\rho$E. A pair of
global and local vector are created using DMCreateGlobalVector and
DMCreateLocalVector. 

I also needed a vector to store the derivatives of the macroscopic
variables, but I have the difficulty to set two PetscSection to a dm
object. The first section has four variables on the cell point, and the
variable number of second section is  not the same as the first
section. I have no 
idea about this problem, and I think that whether I can scatter it by
myself without using  DMGlobalToLocal()?

Can you point me out how can I creat two PetscSection whin a dmplex
object, and how can I using DMGlobalToLocal() for the two pair of
vectors whin a dmplex object? 
> 
> > I have
> > scatter the vectors using VecScatterBegin and VecScatterEnd, and
> the IS
> > is created using the localId and the glolbalId (Obtained by
> > DMPlexGetCellNumbering). I got a wrong result.
> >
> 
> DMPlexGetCellNumbering() gives you are _arbitrary_ numbering of the
> cells,
> not the
> global numbering of unknowns.
> 
> 
> >   Can anyone give me some advices, and does it have some to do with
> the
> > natural ids?
> >
> 
> 1) You can get the local ids from the local Section (DMGetSection),
> and the
> global ids
> from the global Section (DMGetGlobalSection). You loop over all the
> points
> in the mesh
> and get the offset/size pairs. However, this is error-prone (since
> you can
> have constraints)
> and probably unnecessary.
> 
> 2) If you look at the code for DMGlobalToLocalBegin():
> 
> 
https://bitbucket.org/petsc/petsc/src/650136806607101f8eb495880e9efa33f2f75729/src/dm/interface/dm.c#lines-2201
> 
> you see it just call PetscSFBcastBegin(), and there is a similar
> BcastEnd()
> in GlobalToLocalEnd(). Thus all
> we are doing is using the SF directly. That is the same as using a
> Scatter.
> 
>   Thanks,
> 
>    Matt
Thank you very much. I have do it yestoday following the code for
DMGlobalToLocalBegin().
> 
> >   Any helps are approciated. Thanks.
> >
> > leejearl
> >
> >
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their
> experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>






From knepley at gmail.com  Tue Jan 15 08:10:31 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 15 Jan 2019 09:10:31 -0500
Subject: [petsc-users] How Can I scatter a global vector to a local
 vector without using DMGlobaltoLocalBegin/DMGlobaltoLocalEnd?
In-Reply-To: <8579311c1a5e115c00969f06b9ad22888f7b20d1.camel@mail.nwpu.edu.cn>
References: <8579311c1a5e115c00969f06b9ad22888f7b20d1.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4GkSJG7Og4U1tT1SJMiXzVyEkchA-6Nx8KtNbCyZRtB86g@mail.gmail.com>

On Tue, Jan 15, 2019 at 4:51 AM leejearl via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi, Matt
>
> > > Hi, all Petscer:
> > >   I am very sorry for disturbing you for my questions.
> >
> >
> > I like your questions.
> >
> >
> > > I have a
> > > problems as follow.
> > >   1. A DMPlex object dm is created.
> > >   2. A global vector and a local vector are created using the
> > following
> > > routines DMCreateGlobalVector and DMCreateLocalVector.
> > >   3. I have initialed global Vector. Because the local vector have
> > some
> > > overlap cells, I want to scatter the data from global to local.
> > >   4. If I don't use the routines
> > > DMGlobaltoLocalBegin/DMGlobaltoLocalEnd, how can I scatter it?
> >
> >
> > To understand the question better, why would you not use
> > DMGlobalToLocal()?
> I have create a PetscSection for the dm object. The section is used to
> store the macroscopic variables $\rho$, u,  v, $\rho$E. A pair of
> global and local vector are created using DMCreateGlobalVector and
> DMCreateLocalVector.
>
> I also needed a vector to store the derivatives of the macroscopic
> variables, but I have the difficulty to set two PetscSection to a dm
> object.


You are right. You can't use 2 Sections for 1 DM. However, this one is easy.
Use DMClone() to get another DM. They will share the expensive topology
information, but the new one can have the different Section. Then all the
GlobalToLocal will be created automatically.

  Thanks,

    Matt


> The first section has four variables on the cell point, and the
> variable number of second section is  not the same as the first
> section. I have no
> idea about this problem, and I think that whether I can scatter it by
> myself without using  DMGlobalToLocal()?
>
> Can you point me out how can I creat two PetscSection whin a dmplex
> object, and how can I using DMGlobalToLocal() for the two pair of
> vectors whin a dmplex object?
> >
> > > I have
> > > scatter the vectors using VecScatterBegin and VecScatterEnd, and
> > the IS
> > > is created using the localId and the glolbalId (Obtained by
> > > DMPlexGetCellNumbering). I got a wrong result.
> > >
> >
> > DMPlexGetCellNumbering() gives you are _arbitrary_ numbering of the
> > cells,
> > not the
> > global numbering of unknowns.
> >
> >
> > >   Can anyone give me some advices, and does it have some to do with
> > the
> > > natural ids?
> > >
> >
> > 1) You can get the local ids from the local Section (DMGetSection),
> > and the
> > global ids
> > from the global Section (DMGetGlobalSection). You loop over all the
> > points
> > in the mesh
> > and get the offset/size pairs. However, this is error-prone (since
> > you can
> > have constraints)
> > and probably unnecessary.
> >
> > 2) If you look at the code for DMGlobalToLocalBegin():
> >
> >
>
> https://bitbucket.org/petsc/petsc/src/650136806607101f8eb495880e9efa33f2f75729/src/dm/interface/dm.c#lines-2201
> >
> > you see it just call PetscSFBcastBegin(), and there is a similar
> > BcastEnd()
> > in GlobalToLocalEnd(). Thus all
> > we are doing is using the SF directly. That is the same as using a
> > Scatter.
> >
> >   Thanks,
> >
> >    Matt
> Thank you very much. I have do it yestoday following the code for
> DMGlobalToLocalBegin().
> >
> > >   Any helps are approciated. Thanks.
> > >
> > > leejearl
> > >
> > >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> > http://www.cse.buffalo.edu/~knepley/>
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190115/4c6a41e0/attachment.html>

From maahi.buet at gmail.com  Tue Jan 15 11:35:15 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Tue, 15 Jan 2019 12:35:15 -0500
Subject: [petsc-users] PETSc turtorial example generating unexpected
 result
In-Reply-To: <CADOhEh4b=R1oUDrBCbwAuUBph9ESdyLuqRR0wOYg9joz=yXdSA@mail.gmail.com>
References: <CAJOaupFo2dMMh9v8EBsUxuGHoX1LXr3824PHpfQ2oO9LOhxpXw@mail.gmail.com>
	<CADOhEh4b=R1oUDrBCbwAuUBph9ESdyLuqRR0wOYg9joz=yXdSA@mail.gmail.com>
Message-ID: <CAJOaupENkEXySQJLh9Yjbg42VqZs9SWSqetE9eeRYgCcbGAfqA@mail.gmail.com>

Hi,

Thank you for your reply. Yeah, as you suspected, I printed these numbers
after that statement. Now it gives the expected values.

Regards,
Maahi Talukder

On Mon, Jan 14, 2019 at 9:33 PM Mark Adams <mfadams at lbl.gov> wrote:

> This code puts the error in x:
>
> call VecAXPY(x,neg_one,u,ierr)
>
> I suspect you printed these numbers after this statement.
>
> On Mon, Jan 14, 2019 at 8:10 PM Maahi Talukder via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello all,
>>
>> I complied and run the example *ex2f.F90* located in  */home/maahi/petsc/src/ksp/ksp/examples/tutorials.
>> *As in the code the right hand side vector, b, was computed assuming an
>> exact solution of a vector with all elements of 1.0, all the values of
>> vector 'x' should have been 1.0. But after running the code as it was
>> provided, the values of vector x that I get are the followings -
>>  2.72322e-07
>> 3.81437e-07
>> 1.58922e-07
>> 3.81437e-07
>> 2.38878e-07
>> -6.65645e-07
>> 1.58922e-07
>> -6.65645e-07
>> -2.51219e-07
>>
>> I am wondering what went wrong? Could you please help me with some
>> insight into this?
>>
>> Regards,
>> Maahi Talukder
>> Clarkson University
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190115/5bcadc5c/attachment-0001.html>

From leejearl at mail.nwpu.edu.cn  Tue Jan 15 19:15:16 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Wed, 16 Jan 2019 09:15:16 +0800
Subject: [petsc-users] How Can I scatter a global vector to a local vector
 without using DMGlobaltoLocalBegin/DMGlobaltoLocalEnd?
Message-ID: <e7f4e25b7716fa21cd93964460e7381252d12199.camel@mail.nwpu.edu.cn>

Hi Matt,

Thanks for your helps. I have another question? but I think I should
ask the question in a new subject.


leejearl


> > Hi, Matt
> >
> > > > Hi, all Petscer:
> > > >   I am very sorry for disturbing you for my questions.
> > >
> > >
> > > I like your questions.
> > >
> > >
> > > > I have a
> > > > problems as follow.
> > > >   1. A DMPlex object dm is created.
> > > >   2. A global vector and a local vector are created using the
> > > following
> > > > routines DMCreateGlobalVector and DMCreateLocalVector.
> > > >   3. I have initialed global Vector. Because the local vector
> have
> > > some
> > > > overlap cells, I want to scatter the data from global to local.
> > > >   4. If I don't use the routines
> > > > DMGlobaltoLocalBegin/DMGlobaltoLocalEnd, how can I scatter it?
> > >
> > >
> > > To understand the question better, why would you not use
> > > DMGlobalToLocal()?
> > I have create a PetscSection for the dm object. The section is used
> to
> > store the macroscopic variables $\rho$, u,  v, $\rho$E. A pair of
> > global and local vector are created using DMCreateGlobalVector and
> > DMCreateLocalVector.
> >
> > I also needed a vector to store the derivatives of the macroscopic
> > variables, but I have the difficulty to set two PetscSection to a
> dm
> > object.
> 
> 
> You are right. You can't use 2 Sections for 1 DM. However, this one
> is easy.
> Use DMClone() to get another DM. They will share the expensive
> topology
> information, but the new one can have the different Section. Then all
> the
> GlobalToLocal will be created automatically.
> 
>   Thanks,
> 
>     Matt
> 
> 
> > The first section has four variables on the cell point, and the
> > variable number of second section is  not the same as the first
> > section. I have no
> > idea about this problem, and I think that whether I can scatter it
> by
> > myself without using  DMGlobalToLocal()?
> >
> > Can you point me out how can I creat two PetscSection whin a dmplex
> > object, and how can I using DMGlobalToLocal() for the two pair of
> > vectors whin a dmplex object?
> > >
> > > > I have
> > > > scatter the vectors using VecScatterBegin and VecScatterEnd,
> and
> > > the IS
> > > > is created using the localId and the glolbalId (Obtained by
> > > > DMPlexGetCellNumbering). I got a wrong result.
> > > >
> > >
> > > DMPlexGetCellNumbering() gives you are _arbitrary_ numbering of
> the
> > > cells,
> > > not the
> > > global numbering of unknowns.
> > >
> > >
> > > >   Can anyone give me some advices, and does it have some to do
> with
> > > the
> > > > natural ids?
> > > >
> > >
> > > 1) You can get the local ids from the local Section
> (DMGetSection),
> > > and the
> > > global ids
> > > from the global Section (DMGetGlobalSection). You loop over all
> the
> > > points
> > > in the mesh
> > > and get the offset/size pairs. However, this is error-prone
> (since
> > > you can
> > > have constraints)
> > > and probably unnecessary.
> > >
> > > 2) If you look at the code for DMGlobalToLocalBegin():
> > >
> > >
> >
> > 
> https://bitbucket.org/petsc/petsc/src/650136806607101f8eb495880e9efa33f2f75729/src/dm/interface/dm.c#lines-2201
> > >
> > > you see it just call PetscSFBcastBegin(), and there is a similar
> > > BcastEnd()
> > > in GlobalToLocalEnd(). Thus all
> > > we are doing is using the SF directly. That is the same as using
> a
> > > Scatter.
> > >
> > >   Thanks,
> > >
> > >    Matt
> > Thank you very much. I have do it yestoday following the code for
> > DMGlobalToLocalBegin().
> > >
> > > >   Any helps are approciated. Thanks.
> > > >
> > > > leejearl
> > > >
> > > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to
> which
> > > their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/ <
> > > http://www.cse.buffalo.edu/~knepley/>;
> >
> >
> >
> >
> >
> >
> 



From leejearl at mail.nwpu.edu.cn  Tue Jan 15 19:34:29 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Wed, 16 Jan 2019 09:34:29 +0800
Subject: [petsc-users] Is there easy to update the ghost value in the vector
 created using DMCreatLocalVector?
Message-ID: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>

Hi all Petscer,

I have a question about how to update the ghost value in the vector.
The prolblem is as follow.

1. A dmplex dm object is created using DMPlexCreateFromFile
2. The dm is distributed using DMPlexDistribute, and the overlap is set
as 1.
3. The ghosted dm is constructed by DMConstructGhostCells.
4. A global Vector X and a local vector localX are created by
DMCreateGlobalVector and DMCreateLocalVector

Now, the problem is that the ghost value are needed in the computation
using local vector localX. and the ghost value must be update from the
global vector. 

For example, gCellId is the index of ghost cell in the local vector
localX of process 0, and cellId is the index of cell in the global
vector of process 1. I must update the data of cellId to gCellId. How
can I do it easily?

I can implement it use the PetscSFBcastBegin/PetscSFBcastEnd. Is there
other simple way to do it?


Any helps are appreciated.

Thanks.



leejearl


From tempohoper at gmail.com  Wed Jan 16 02:51:07 2019
From: tempohoper at gmail.com (Sal Am)
Date: Wed, 16 Jan 2019 08:51:07 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
Message-ID: <CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>

>
> The memory requested is an insane number. You may need to use 64 bit
> integers.

Thanks Mark, I reconfigured it to use 64bit, however in the process it says
I can no longer use MUMPS and SuperLU as they are not supported (I see on
MUMPS webpage it supports 64int). However it does not exactly solve the
problem.

This time, it crashes at
> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989
> in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> ierr      = PetscMalloc1(bi[pn]+1,&bj);
> which allocates local portion of B^T*A.
> You may also try to increase number of cores to reduce local matrix size.
>

So I increased the number of cores to 16 on one node and ran it by :
mpiexec valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
-pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
It crashed after reading in the matrix and before starting to solve. The
error:

[15]PETSC ERROR: [0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
[0]PETSC ERROR: [1]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
[3]PETSC ERROR:
------------------------------------------------------------------------
[4]PETSC ERROR:
------------------------------------------------------------------------
[4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
------------------------------------------------------------------------
[8]PETSC ERROR:
------------------------------------------------------------------------
[12]PETSC ERROR:
------------------------------------------------------------------------
[12]PETSC ERROR: [14]PETSC ERROR:
------------------------------------------------------------------------
[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
--------------------------------------------------------------------------
mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on
signal 9 (Killed).

Now I was running this with valgrind as someone had previously suggested
and the 16 files created all contain the same type of error:

==25940== Invalid read of size 8
==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:373)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16 alloc'd
==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:371)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==


On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Fande:
>
>> According to this PR
>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>
>> Should we set the scalable algorithm as default?
>>
> Sure, we can. But I feel we need do more tests to compare scalable and
> non-scalable algorithms.
> On theory, for small to medium matrices, non-scalable matmatmult()
> algorithm enables more efficient
> data accessing. Andreas optimized scalable implementation. Our
> non-scalable implementation might have room to be further optimized.
> Hong
>
>>
>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Add option '-mattransposematmult_via scalable'
>>> Hong
>>>
>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> I saw the following error message in your first email.
>>>>
>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>
>>>> Probably the matrix is too large. You can try with more compute nodes,
>>>> for example, use 8 nodes instead of 2, and see what happens.
>>>>
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Using a larger problem set with 2B non-zero elements and a matrix of
>>>>> 25M x 25M I get the following error:
>>>>> [4]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>>>> probably memory access out of range
>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> [4]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>>>> OS X to find memory corruption errors
>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>> ------------------------------------
>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>> available,
>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>> function
>>>>> [4]PETSC ERROR:       is given.
>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>> [4]PETSC ERROR: [4]
>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [4]PETSC ERROR: Signal received
>>>>> [4]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>>>> vef002 Fri Jan 11 09:13:23 2019
>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>> --download-fblaslapack=1 --download-cmake
>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>
>>>>> --------------------------------------------------------------------------
>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>> with errorcode 59.
>>>>>
>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>> You may or may not see output from other processes, depending on
>>>>> exactly when Open MPI kills them.
>>>>>
>>>>> --------------------------------------------------------------------------
>>>>> [0]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>> the batch system) has told this process to end
>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> [0]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>
>>>>> Using Valgrind on only one of the valgrind files the following error
>>>>> was written:
>>>>>
>>>>> ==9053== Invalid read of size 4
>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>> (matmatmult.c:790)
>>>>> ==9053==    by 0x5D106F8:
>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>> (mpimatmatmult.c:1186)
>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>>>> free'd
>>>>> ==9053==
>>>>>
>>>>>
>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>
>>>>>> Thank you Dave,
>>>>>>
>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the code
>>>>>> again with the following options:
>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>> gamg -log_view
>>>>>> (as on the petsc website you linked)
>>>>>>
>>>>>> It finished solving using the iterative solver, but the resulting
>>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>
>>>>>>>
>>>>>>> This likely implies that you have a memory error in your code (a
>>>>>>> memory leak would not cause this behaviour).
>>>>>>> I strongly suggest you make sure your code is free of memory errors.
>>>>>>> You can do this using valgrind. See here
>>>>>>>
>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>
>>>>>>> for an explanation of how to use valgrind.
>>>>>>>
>>>>>>>
>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>
>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>
>>>>>>>> how I run it in the job script on 2 node with 32 processors using
>>>>>>>> the clusters OpenMPI.
>>>>>>>>
>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>
>>>>>>>> the matrix:
>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>
>>>>>>>> Thanks and all the best
>>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/81f50a85/attachment-0001.html>

From colin.cotter at imperial.ac.uk  Wed Jan 16 03:32:17 2019
From: colin.cotter at imperial.ac.uk (Cotter, Colin J)
Date: Wed, 16 Jan 2019 09:32:17 +0000
Subject: [petsc-users] SLEPc: Setting options for mass matrix KSP in
 generalised eigenvalue problem
Message-ID: <VI1PR0601MB2142A4B866290467F7EE2C01B3820@VI1PR0601MB2142.eurprd06.prod.outlook.com>

Dear PETSc mailing list,

  A SLEPc question, sorry if this is obvious and I missed something.


How do I set the KSP for the mass matrix M in the generalised eigenvalue problem

(I'm looking for the eigenvalue with largest magnitude)


Ax = lambda*Mx


?

I can only find how to set the KSP for a spectral transform.


(I'm actually using slepc4py)


all the best

-_Colin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/03d58d19/attachment.html>

From jroman at dsic.upv.es  Wed Jan 16 03:55:08 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 16 Jan 2019 10:55:08 +0100
Subject: [petsc-users] SLEPc: Setting options for mass matrix KSP in
 generalised eigenvalue problem
In-Reply-To: <VI1PR0601MB2142A4B866290467F7EE2C01B3820@VI1PR0601MB2142.eurprd06.prod.outlook.com>
References: <VI1PR0601MB2142A4B866290467F7EE2C01B3820@VI1PR0601MB2142.eurprd06.prod.outlook.com>
Message-ID: <715B26BA-6154-49FB-A733-E6CB593CF0A2@dsic.upv.es>

This case is treated in the same way as the (shift-and-invert) spectral transformation. There are nested objects EPS > ST > KSP > PC. By default ST=SHIFT, which is what you need. So you just have to get access to the KSP/PC objects to set options.

Basically you can use ex9.py in slepc4py but use F.setType(F.Type.SHIFT) instead of F.setType(F.Type.PRECOND).

Jose


> El 16 ene 2019, a las 10:32, Cotter, Colin J via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Dear PETSc mailing list,
>   A SLEPc question, sorry if this is obvious and I missed something.
> 
> How do I set the KSP for the mass matrix M in the generalised eigenvalue problem
> (I'm looking for the eigenvalue with largest magnitude)
> 
> Ax = lambda*Mx
> 
> ?
> 
> I can only find how to set the KSP for a spectral transform. 
> 
> (I'm actually using slepc4py)
> 
> all the best
> -_Colin


From knepley at gmail.com  Wed Jan 16 07:07:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Jan 2019 08:07:04 -0500
Subject: [petsc-users] Is there easy to update the ghost value in the
 vector created using DMCreatLocalVector?
In-Reply-To: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>
References: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4GnJ+d33Rx-QYr6hHS+7B_xne23=LZGuBk_RfzKm2qjdbA@mail.gmail.com>

On Tue, Jan 15, 2019 at 8:34 PM leejearl via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi all Petscer,
>
> I have a question about how to update the ghost value in the vector.
> The prolblem is as follow.
>
> 1. A dmplex dm object is created using DMPlexCreateFromFile
> 2. The dm is distributed using DMPlexDistribute, and the overlap is set
> as 1.
> 3. The ghosted dm is constructed by DMConstructGhostCells.
> 4. A global Vector X and a local vector localX are created by
> DMCreateGlobalVector and DMCreateLocalVector
>
> Now, the problem is that the ghost value are needed in the computation
> using local vector localX. and the ghost value must be update from the
> global vector.
>
> For example, gCellId is the index of ghost cell in the local vector
> localX of process 0, and cellId is the index of cell in the global
> vector of process 1. I must update the data of cellId to gCellId. How
> can I do it easily?
>
> I can implement it use the PetscSFBcastBegin/PetscSFBcastEnd. Is there
> other simple way to do it?
>

Doesn't DMGlobalToLocal() work?

  Thanks,

    Matt


> Any helps are appreciated.
>
> Thanks.
>
>
>
> leejearl
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/f96af7b7/attachment.html>

From colin.cotter at imperial.ac.uk  Wed Jan 16 07:08:53 2019
From: colin.cotter at imperial.ac.uk (Cotter, Colin J)
Date: Wed, 16 Jan 2019 13:08:53 +0000
Subject: [petsc-users] SLEPc: Setting options for mass matrix KSP in
 generalised eigenvalue problem
In-Reply-To: <715B26BA-6154-49FB-A733-E6CB593CF0A2@dsic.upv.es>
References: <VI1PR0601MB2142A4B866290467F7EE2C01B3820@VI1PR0601MB2142.eurprd06.prod.outlook.com>,
	<715B26BA-6154-49FB-A733-E6CB593CF0A2@dsic.upv.es>
Message-ID: <VI1PR0601MB2142E8A3D4D9C829817594C4B3820@VI1PR0601MB2142.eurprd06.prod.outlook.com>

Thanks Jose!


all the best

--cjc

________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: 16 January 2019 09:55:08
To: Cotter, Colin J
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] SLEPc: Setting options for mass matrix KSP in generalised eigenvalue problem

This case is treated in the same way as the (shift-and-invert) spectral transformation. There are nested objects EPS > ST > KSP > PC. By default ST=SHIFT, which is what you need. So you just have to get access to the KSP/PC objects to set options.

Basically you can use ex9.py in slepc4py but use F.setType(F.Type.SHIFT) instead of F.setType(F.Type.PRECOND).

Jose


> El 16 ene 2019, a las 10:32, Cotter, Colin J via petsc-users <petsc-users at mcs.anl.gov> escribi?:
>
> Dear PETSc mailing list,
>   A SLEPc question, sorry if this is obvious and I missed something.
>
> How do I set the KSP for the mass matrix M in the generalised eigenvalue problem
> (I'm looking for the eigenvalue with largest magnitude)
>
> Ax = lambda*Mx
>
> ?
>
> I can only find how to set the KSP for a spectral transform.
>
> (I'm actually using slepc4py)
>
> all the best
> -_Colin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/438336f8/attachment.html>

From knepley at gmail.com  Wed Jan 16 07:12:13 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Jan 2019 08:12:13 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
Message-ID: <CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>

On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> The memory requested is an insane number. You may need to use 64 bit
>> integers.
>
> Thanks Mark, I reconfigured it to use 64bit, however in the process it
> says I can no longer use MUMPS and SuperLU as they are not supported (I see
> on MUMPS webpage it supports 64int). However it does not exactly solve the
> problem.
>

SuperLU_dist supports 64-bit ints. Are you not running in parallel?


> This time, it crashes at
>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989
>> in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>> which allocates local portion of B^T*A.
>> You may also try to increase number of cores to reduce local matrix size.
>>
>
> So I increased the number of cores to 16 on one node and ran it by :
> mpiexec valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
> It crashed after reading in the matrix and before starting to solve. The
> error:
>
> [15]PETSC ERROR: [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [3]PETSC ERROR:
> ------------------------------------------------------------------------
> [4]PETSC ERROR:
> ------------------------------------------------------------------------
> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
> ------------------------------------------------------------------------
> [8]PETSC ERROR:
> ------------------------------------------------------------------------
> [12]PETSC ERROR:
> ------------------------------------------------------------------------
> [12]PETSC ERROR: [14]PETSC ERROR:
> ------------------------------------------------------------------------
> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> --------------------------------------------------------------------------
> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on
> signal 9 (Killed).
>
> Now I was running this with valgrind as someone had previously suggested
> and the 16 files created all contain the same type of error:
>

Okay, its possible that there are bugs in the MPI implementation. So

1) Try using    -build_twosided allreduce    on this run

2) Is it possible to get something that fails here but we can run. None of
our tests show this problem.

  Thanks,

     Matt


> ==25940== Invalid read of size 8
> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
> (mpits.c:373)
> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16 alloc'd
> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
> (mpits.c:371)
> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
> ==25940==
>
>
> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Fande:
>>
>>> According to this PR
>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>
>>> Should we set the scalable algorithm as default?
>>>
>> Sure, we can. But I feel we need do more tests to compare scalable and
>> non-scalable algorithms.
>> On theory, for small to medium matrices, non-scalable matmatmult()
>> algorithm enables more efficient
>> data accessing. Andreas optimized scalable implementation. Our
>> non-scalable implementation might have room to be further optimized.
>> Hong
>>
>>>
>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Add option '-mattransposematmult_via scalable'
>>>> Hong
>>>>
>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> I saw the following error message in your first email.
>>>>>
>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>
>>>>> Probably the matrix is too large. You can try with more compute nodes,
>>>>> for example, use 8 nodes instead of 2, and see what happens.
>>>>>
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Using a larger problem set with 2B non-zero elements and a matrix of
>>>>>> 25M x 25M I get the following error:
>>>>>> [4]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>>>>> probably memory access out of range
>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>> -on_error_attach_debugger
>>>>>> [4]PETSC ERROR: or see
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>> Mac OS X to find memory corruption errors
>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>> ------------------------------------
>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>> available,
>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>>> function
>>>>>> [4]PETSC ERROR:       is given.
>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>> [4]PETSC ERROR: [4]
>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>> [4]PETSC ERROR: Signal received
>>>>>> [4]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>>>>> vef002 Fri Jan 11 09:13:23 2019
>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>> with errorcode 59.
>>>>>>
>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>> You may or may not see output from other processes, depending on
>>>>>> exactly when Open MPI kills them.
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> [0]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>> the batch system) has told this process to end
>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>> -on_error_attach_debugger
>>>>>> [0]PETSC ERROR: or see
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>
>>>>>> Using Valgrind on only one of the valgrind files the following error
>>>>>> was written:
>>>>>>
>>>>>> ==9053== Invalid read of size 4
>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>> (matmatmult.c:790)
>>>>>> ==9053==    by 0x5D106F8:
>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>> (mpimatmatmult.c:1186)
>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>>>>> free'd
>>>>>> ==9053==
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>>
>>>>>>> Thank you Dave,
>>>>>>>
>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the
>>>>>>> code again with the following options:
>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>> gamg -log_view
>>>>>>> (as on the petsc website you linked)
>>>>>>>
>>>>>>> It finished solving using the iterative solver, but the resulting
>>>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>
>>>>>>>>
>>>>>>>> This likely implies that you have a memory error in your code (a
>>>>>>>> memory leak would not cause this behaviour).
>>>>>>>> I strongly suggest you make sure your code is free of memory errors.
>>>>>>>> You can do this using valgrind. See here
>>>>>>>>
>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>
>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>
>>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>>
>>>>>>>>> how I run it in the job script on 2 node with 32 processors using
>>>>>>>>> the clusters OpenMPI.
>>>>>>>>>
>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>
>>>>>>>>> the matrix:
>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>
>>>>>>>>> Thanks and all the best
>>>>>>>>>
>>>>>>>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/8cce6c56/attachment-0001.html>

From leejearl at mail.nwpu.edu.cn  Wed Jan 16 07:52:33 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Wed, 16 Jan 2019 21:52:33 +0800
Subject: [petsc-users] Is there easy to update the ghost value in the
 vector created using DMCreatLocalVector?
In-Reply-To: <CAMYG4GnJ+d33Rx-QYr6hHS+7B_xne23=LZGuBk_RfzKm2qjdbA@mail.gmail.com>
References: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>
	<CAMYG4GnJ+d33Rx-QYr6hHS+7B_xne23=LZGuBk_RfzKm2qjdbA@mail.gmail.com>
Message-ID: <00243785c19f461c4826d48f5efe5040a3d124e6.camel@mail.nwpu.edu.cn>

Hi Matt,	Thanks for you reply.  If I use  DMGlobalToLocal() to
update the local vector from global vector.The value of
cStart<=c<cEndinterior is ok. But the value of cEndInterior<=c<cEnd is
not the right value.	I do not know whether it has something to do
with the DMGlobalToLocal().  My problem is as follow.After the mesh is
distributed , the cells and ghost cells (local vector) look like
processor 0: 0, 1, 2, 3, 4 (cell 4 is the ghost cell)processor 1: 0, 1,
2, 3, 4 (cell 4 is the ghost cell)
The value of cell 4 in "processor 0" is same as the value of  cell 0 in
"processor 1". The value of cell 0 in  "processor 1" hasbeen set in the
global vector. I should copy  the corresponding global value of cell 0
in  "processor 1"  to cell 4 in  "processor 0".
This is my problem. I want to know how to implement it easily. Can you
give me some advices?
Thanks
leejearl

On Wed, 2019-01-16 at 08:07 -0500, Matthew Knepley wrote:
> On Tue, Jan 15, 2019 at 8:34 PM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi all Petscer,
> > 
> > 
> > 
> > I have a question about how to update the ghost value in the
> > vector.
> > 
> > The prolblem is as follow.
> > 
> > 
> > 
> > 1. A dmplex dm object is created using DMPlexCreateFromFile
> > 
> > 2. The dm is distributed using DMPlexDistribute, and the overlap is
> > set
> > 
> > as 1.
> > 
> > 3. The ghosted dm is constructed by DMConstructGhostCells.
> > 
> > 4. A global Vector X and a local vector localX are created by
> > 
> > DMCreateGlobalVector and DMCreateLocalVector
> > 
> > 
> > 
> > Now, the problem is that the ghost value are needed in the
> > computation
> > 
> > using local vector localX. and the ghost value must be update from
> > the
> > 
> > global vector. 
> > 
> > 
> > 
> > For example, gCellId is the index of ghost cell in the local vector
> > 
> > localX of process 0, and cellId is the index of cell in the global
> > 
> > vector of process 1. I must update the data of cellId to gCellId.
> > How
> > 
> > can I do it easily?
> > 
> > 
> > 
> > I can implement it use the PetscSFBcastBegin/PetscSFBcastEnd. Is
> > there
> > 
> > other simple way to do it?
> 
> Doesn't DMGlobalToLocal() work?
>   Thanks,
>     Matt 
> > Any helps are appreciated.
> > 
> > 
> > 
> > Thanks.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > leejearl
> > 
> > 
> > 
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/fd4cc9e4/attachment.html>

From knepley at gmail.com  Wed Jan 16 08:15:51 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Jan 2019 09:15:51 -0500
Subject: [petsc-users] Is there easy to update the ghost value in the
 vector created using DMCreatLocalVector?
In-Reply-To: <00243785c19f461c4826d48f5efe5040a3d124e6.camel@mail.nwpu.edu.cn>
References: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>
	<CAMYG4GnJ+d33Rx-QYr6hHS+7B_xne23=LZGuBk_RfzKm2qjdbA@mail.gmail.com>
	<00243785c19f461c4826d48f5efe5040a3d124e6.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4G=vk1yLdcvF_Lj9zbzfJEwHJhSuOe_t+yJJL0QdE3S3Eg@mail.gmail.com>

On Wed, Jan 16, 2019 at 8:52 AM leejearl <leejearl at mail.nwpu.edu.cn> wrote:

> Hi Matt,
> Thanks for you reply. If I use DMGlobalToLocal() to update the local
> vector from global vector.
> The value of cStart<=c<cEndinterior is ok. But the value of
> cEndInterior<=c<cEnd is not the right value.
> I do not know whether it has something to do with the DMGlobalToLocal().
> My problem is as follow.
> After the mesh is distributed , the cells and ghost cells (local vector)
> look like
>

Hmm, that sounds like a bug, but I have to think a little bit. I don't see
how TS ex11 can be working in parallel
if overlapping cells are not being updated. We should make a small example
so that we are certainly comparing the
same things.

  Thanks,

    Matt


> processor 0: 0, 1, 2, 3, 4 (cell 4 is the ghost cell)
> processor 1: 0, 1, 2, 3, 4 (cell 4 is the ghost cell)
>
> The value of cell 4 in "processor 0" is same as the value of cell 0 in
> "processor 1". The value of cell 0 in "processor 1" has
> been set in the global vector. I should copy the corresponding global
> value of cell 0 in "processor 1" to cell 4 in "processor 0".
>
> This is my problem. I want to know how to implement it easily. Can you
> give me some advices?
>
> Thanks
>
> leejearl
>
>
> On Wed, 2019-01-16 at 08:07 -0500, Matthew Knepley wrote:
>
> On Tue, Jan 15, 2019 at 8:34 PM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi all Petscer,
>
> I have a question about how to update the ghost value in the vector.
> The prolblem is as follow.
>
> 1. A dmplex dm object is created using DMPlexCreateFromFile
> 2. The dm is distributed using DMPlexDistribute, and the overlap is set
> as 1.
> 3. The ghosted dm is constructed by DMConstructGhostCells.
> 4. A global Vector X and a local vector localX are created by
> DMCreateGlobalVector and DMCreateLocalVector
>
> Now, the problem is that the ghost value are needed in the computation
> using local vector localX. and the ghost value must be update from the
> global vector.
>
> For example, gCellId is the index of ghost cell in the local vector
> localX of process 0, and cellId is the index of cell in the global
> vector of process 1. I must update the data of cellId to gCellId. How
> can I do it easily?
>
> I can implement it use the PetscSFBcastBegin/PetscSFBcastEnd. Is there
> other simple way to do it?
>
>
> Doesn't DMGlobalToLocal() work?
>
>   Thanks,
>
>     Matt
>
>
> Any helps are appreciated.
>
> Thanks.
>
>
>
> leejearl
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/b83ea21c/attachment.html>

From leejearl at mail.nwpu.edu.cn  Wed Jan 16 08:43:15 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Wed, 16 Jan 2019 22:43:15 +0800
Subject: [petsc-users] Is there easy to update the ghost value in the
 vector created using DMCreatLocalVector?
In-Reply-To: <CAMYG4G=vk1yLdcvF_Lj9zbzfJEwHJhSuOe_t+yJJL0QdE3S3Eg@mail.gmail.com>
References: <ae596f9506ae815add812886a687a97ba4b95429.camel@mail.nwpu.edu.cn>
	<CAMYG4GnJ+d33Rx-QYr6hHS+7B_xne23=LZGuBk_RfzKm2qjdbA@mail.gmail.com>
	<00243785c19f461c4826d48f5efe5040a3d124e6.camel@mail.nwpu.edu.cn>
	<CAMYG4G=vk1yLdcvF_Lj9zbzfJEwHJhSuOe_t+yJJL0QdE3S3Eg@mail.gmail.com>
Message-ID: <0fe2a8f9ab455af95c1a818f6a0616b52216e372.camel@mail.nwpu.edu.cn>

Hi Matt?	I think it is not a bug. DMGlobaltoLocal works well.
Maybe I have not explain my problem clearly.I will detail it
tomorrow.       Thanks?leejearl

On Wed, 2019-01-16 at 09:15 -0500, Matthew Knepley wrote:
> On Wed, Jan 16, 2019 at 8:52 AM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
> > Hi Matt,
> > 	Thanks for you reply.  If I use  DMGlobalToLocal() to update
> > the local vector from global vector.
> > The value of cStart<=c<cEndinterior is ok. But the value of
> > cEndInterior<=c<cEnd is not the right value.
> > 	I do not know whether it has something to do with the
> > DMGlobalToLocal().  My problem is as follow.
> > After the mesh is distributed , the cells and ghost cells (local
> > vector) look like
> 
> Hmm, that sounds like a bug, but I have to think a little bit. I
> don't see how TS ex11 can be working in parallelif overlapping cells
> are not being updated. We should make a small example so that we are
> certainly comparing thesame things.
>   Thanks,
>     Matt 
> > processor 0: 0, 1, 2, 3, 4 (cell 4 is the ghost cell)
> > processor 1: 0, 1, 2, 3, 4 (cell 4 is the ghost cell)
> > 
> > The value of cell 4 in "processor 0" is same as the value of  cell
> > 0 in "processor 1". The value of cell 0 in  "processor 1" has
> > been set in the global vector. I should copy  the corresponding
> > global value of cell 0 in  "processor 1"  to cell 4 in  "processor
> > 0".
> > 
> > This is my problem. I want to know how to implement it easily. Can
> > you give me some advices?
> > 
> > Thanks
> > 
> > leejearl
> > 
> > 
> > On Wed, 2019-01-16 at 08:07 -0500, Matthew Knepley wrote:
> > > On Tue, Jan 15, 2019 at 8:34 PM leejearl via petsc-users <
> > > petsc-users at mcs.anl.gov> wrote:
> > > > Hi all Petscer,
> > > > 
> > > > 
> > > > 
> > > > I have a question about how to update the ghost value in the
> > > > vector.
> > > > 
> > > > The prolblem is as follow.
> > > > 
> > > > 
> > > > 
> > > > 1. A dmplex dm object is created using DMPlexCreateFromFile
> > > > 
> > > > 2. The dm is distributed using DMPlexDistribute, and the
> > > > overlap is set
> > > > 
> > > > as 1.
> > > > 
> > > > 3. The ghosted dm is constructed by DMConstructGhostCells.
> > > > 
> > > > 4. A global Vector X and a local vector localX are created by
> > > > 
> > > > DMCreateGlobalVector and DMCreateLocalVector
> > > > 
> > > > 
> > > > 
> > > > Now, the problem is that the ghost value are needed in the
> > > > computation
> > > > 
> > > > using local vector localX. and the ghost value must be update
> > > > from the
> > > > 
> > > > global vector. 
> > > > 
> > > > 
> > > > 
> > > > For example, gCellId is the index of ghost cell in the local
> > > > vector
> > > > 
> > > > localX of process 0, and cellId is the index of cell in the
> > > > global
> > > > 
> > > > vector of process 1. I must update the data of cellId to
> > > > gCellId. How
> > > > 
> > > > can I do it easily?
> > > > 
> > > > 
> > > > 
> > > > I can implement it use the PetscSFBcastBegin/PetscSFBcastEnd.
> > > > Is there
> > > > 
> > > > other simple way to do it?
> > > 
> > > Doesn't DMGlobalToLocal() work?
> > >   Thanks,
> > >     Matt 
> > > > Any helps are appreciated.
> > > > 
> > > > 
> > > > 
> > > > Thanks.
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > leejearl
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/dfb423e6/attachment.html>

From yann.jobic at univ-amu.fr  Wed Jan 16 12:12:03 2019
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Wed, 16 Jan 2019 19:12:03 +0100
Subject: [petsc-users] ghost nodes with a DMPlex
Message-ID: <5aacc55d-d43b-c0bf-d803-6373d2c24cbd@univ-amu.fr>

Hi, sorry if this is an obvious question, but i don't know how to use 
the ghost nodes with a DMPlex.

The procedure i do is as follow :

1) create the DMPlex

2) distribute it

3) Create the Matrix and associated vectors with DMCreateMatrix, 
DMCreateGlobalVector and DMCreateLocalVector

4) In each cells, compute the different integrals (FEM code)

I know the location of the ghost points, as the global index is negative 
(and the global index is -(i+1) using the global PetscSection).

(I didn't use DMPlexConstructGhostCells, as i understood that is should 
be used for flux calculations on faces for FV methods).

I would like to update the contributions of those ghost nodes, where the 
vertex actually is (in a different processor).

So far, i used VecSetValues with the local indexes of the local vector. 
Then i used DMLocalToGlobalBegin in order to update them. It's working fine.

I tried? MatSetValues on the global Matrix with the global indexes. It 
seems to work.

However, maybe i'm doing it the wrong way, with a lot of communications. 
Is it a scalable way the handle them ? How should i do it ?

If i can have a starting point, it would be great !

Any help would be greatly appreciated!

Regards,

Yann


From knepley at gmail.com  Wed Jan 16 12:15:47 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Jan 2019 13:15:47 -0500
Subject: [petsc-users] ghost nodes with a DMPlex
In-Reply-To: <5aacc55d-d43b-c0bf-d803-6373d2c24cbd@univ-amu.fr>
References: <5aacc55d-d43b-c0bf-d803-6373d2c24cbd@univ-amu.fr>
Message-ID: <CAMYG4G=KfgHgD0kQ9cQW=H0r2OzGG32EynzGnYbaeSoOPDrehw@mail.gmail.com>

On Wed, Jan 16, 2019 at 1:12 PM Yann Jobic via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi, sorry if this is an obvious question, but i don't know how to use
> the ghost nodes with a DMPlex.
>

DMGlobalToLocal() and DMLocalToGlobal().


> The procedure i do is as follow :
>
> 1) create the DMPlex
>
> 2) distribute it
>
> 3) Create the Matrix and associated vectors with DMCreateMatrix,
> DMCreateGlobalVector and DMCreateLocalVector
>
> 4) In each cells, compute the different integrals (FEM code)
>

Yes


> I know the location of the ghost points, as the global index is negative
> (and the global index is -(i+1) using the global PetscSection).
>

Yes


> (I didn't use DMPlexConstructGhostCells, as i understood that is should
> be used for flux calculations on faces for FV methods).
>

Yes


> I would like to update the contributions of those ghost nodes, where the
> vertex actually is (in a different processor).
>

Yes, use LocalToGlobal()


> So far, i used VecSetValues with the local indexes of the local vector.
> Then i used DMLocalToGlobalBegin in order to update them. It's working
> fine.
>

Okay, that is the way to do it.


> I tried  MatSetValues on the global Matrix with the global indexes. It
> seems to work.
>

Yes


> However, maybe i'm doing it the wrong way, with a lot of communications.
> Is it a scalable way the handle them ? How should i do it ?
>

What you described is scalable. There is only 1 communication.

  Thanks,

    Matt


> If i can have a starting point, it would be great !
>
> Any help would be greatly appreciated!
>
> Regards,
>
> Yann
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/82f0f494/attachment.html>

From yjwu16 at gmail.com  Wed Jan 16 21:58:32 2019
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Wed, 16 Jan 2019 22:58:32 -0500
Subject: [petsc-users] Problems about SNES
Message-ID: <CAAMEhBguE9ekWNwZnmm5YfNjiYCamvmhKAmsY3dqnRrLEsp4yg@mail.gmail.com>

Dear PETSc developers:
Hi,
During the process of testing the program, I found some questions about
SNES. These are some basic questions that I have overlooked. Please help me
to answer them.
1. Because my program uses - snes_mf_operator, there is no Jacobian matrix.
Linear and non-linear step residuals are different in petsc. The linear
step residuals are r_linear = J*?x-f(x). Since I don't have a Jacobian
matrix, I don't know how to calculate the relative residuals of linear
steps provided in petsc. Do we use the finite difference approximation
matrix vector product when calculating the residuals?
2. Read the user's manual for a brief introduction to the inexact Newton
method, but I am very interested in the use of this method. I want to know
how to use this method in petsc.
3. The default line search used by SNES in PETSc is bt, which often fails
in program debugging. I don't know much about linesearch, and I'm curious
to know why it failed. How can I supplement this knowledge?

Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190116/dca1e882/attachment.html>

From leejearl at mail.nwpu.edu.cn  Thu Jan 17 02:34:08 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Thu, 17 Jan 2019 16:34:08 +0800
Subject: [petsc-users] I suggest to create a demo code for the problem of
 "Get a value from global vector to a local processor"
Message-ID: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>

Hi all Petscer,
	I have ask helps for some questions. Thanks for the replies
from developer. I have known more about the petsc. I have also sought
helps in the mail lists, and I find that there are many subjects
focused on such a problem. Some subjects are listed as follow


https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
,


https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
,


https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
,


https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html

......

The problem can be summarized as
1. Getting the value of vector from other processor
2. How to set the value of ghost cell, the value of the ghost cell
might be as same as a very cell in other processor. 

I think the above problem is very popular. I suffer such a problem when
I want to treat the periodic boundary for the FVM method. After the dm
object is distributed, the donor cell of bound might be in other
processor. Since the donor cell must be matched as a pair correctly,
using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get the
expected results. 

In fact, such a problem is not very difficult for a mpi program. But we
are coding with the petsc, we always think that whether we can
implement more easily.

Whether our developer can create a demo for us to following. I think
such a demo is very useful for user.

Thanks

leejeal






From knepley at gmail.com  Thu Jan 17 05:43:30 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 06:43:30 -0500
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>

On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi all Petscer,
>         I have ask helps for some questions. Thanks for the replies
> from developer. I have known more about the petsc. I have also sought
> helps in the mail lists, and I find that there are many subjects
> focused on such a problem. Some subjects are listed as follow
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> ,
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
>
> ......
>
> The problem can be summarized as
> 1. Getting the value of vector from other processor
> 2. How to set the value of ghost cell, the value of the ghost cell
> might be as same as a very cell in other processor.
>
> I think the above problem is very popular. I suffer such a problem when
> I want to treat the periodic boundary for the FVM method.


Ah, I think I am now seeing the problem. In DMPlex, we do not implement
periodicity of the mesh by putting in extra communication. We make a
periodic mesh directly. For example, in 1D what you describe would mesh
a line segment, and then communicate the values from one end to the other.
Instead, in Plex we just mesh a circle.

  Thanks,

    Matt


> After the dm
> object is distributed, the donor cell of bound might be in other
> processor. Since the donor cell must be matched as a pair correctly,
> using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get the
> expected results.
>
> In fact, such a problem is not very difficult for a mpi program. But we
> are coding with the petsc, we always think that whether we can
> implement more easily.
>
> Whether our developer can create a demo for us to following. I think
> such a demo is very useful for user.
>
> Thanks
>
> leejeal
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/9475b378/attachment.html>

From tempohoper at gmail.com  Thu Jan 17 07:16:26 2019
From: tempohoper at gmail.com (Sal Am)
Date: Thu, 17 Jan 2019 13:16:26 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
Message-ID: <CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>

>
> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>
I will try that, although I think solving the real problem (later on if I
can get this to work) with 30 million finite elements might be a problem
for SuperLU_dist. so it is better to get an iterative solver to work with
first.

1) Try using    -build_twosided allreduce    on this run
>
How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
-pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
-ksp_monitor -log_view
I have attached the full error output.

2) Is it possible to get something that fails here but we can run. None of
> our tests show this problem.
>
I am not how I can do that, but i have added my code which is quite short
and should only read and solve the system, the problem arises at larger
matrices for example current test case has 6 million finite elements (~2B
non-zero numbers and 25M x 25M matrix).

On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> The memory requested is an insane number. You may need to use 64 bit
>>> integers.
>>
>> Thanks Mark, I reconfigured it to use 64bit, however in the process it
>> says I can no longer use MUMPS and SuperLU as they are not supported (I see
>> on MUMPS webpage it supports 64int). However it does not exactly solve the
>> problem.
>>
>
> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>
>
>> This time, it crashes at
>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989
>>> in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>> which allocates local portion of B^T*A.
>>> You may also try to increase number of cores to reduce local matrix size.
>>>
>>
>> So I increased the number of cores to 16 on one node and ran it by :
>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>> It crashed after reading in the matrix and before starting to solve. The
>> error:
>>
>> [15]PETSC ERROR: [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>> batch system) has told this process to end
>> [0]PETSC ERROR: [1]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [2]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>> batch system) has told this process to end
>> [3]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [4]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [8]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [12]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [12]PETSC ERROR: [14]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>> batch system) has told this process to end
>> --------------------------------------------------------------------------
>> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on
>> signal 9 (Killed).
>>
>> Now I was running this with valgrind as someone had previously suggested
>> and the 16 files created all contain the same type of error:
>>
>
> Okay, its possible that there are bugs in the MPI implementation. So
>
> 1) Try using    -build_twosided allreduce    on this run
>
> 2) Is it possible to get something that fails here but we can run. None of
> our tests show this problem.
>
>   Thanks,
>
>      Matt
>
>
>> ==25940== Invalid read of size 8
>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>> (mpits.c:373)
>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>> alloc'd
>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>> (mpits.c:371)
>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>> ==25940==
>>
>>
>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Fande:
>>>
>>>> According to this PR
>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>
>>>> Should we set the scalable algorithm as default?
>>>>
>>> Sure, we can. But I feel we need do more tests to compare scalable and
>>> non-scalable algorithms.
>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>> algorithm enables more efficient
>>> data accessing. Andreas optimized scalable implementation. Our
>>> non-scalable implementation might have room to be further optimized.
>>> Hong
>>>
>>>>
>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Add option '-mattransposematmult_via scalable'
>>>>> Hong
>>>>>
>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> I saw the following error message in your first email.
>>>>>>
>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>
>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>
>>>>>> --Junchao Zhang
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Using a larger problem set with 2B non-zero elements and a matrix of
>>>>>>> 25M x 25M I get the following error:
>>>>>>> [4]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>> Violation, probably memory access out of range
>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>> -on_error_attach_debugger
>>>>>>> [4]PETSC ERROR: or see
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>> Mac OS X to find memory corruption errors
>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>> ------------------------------------
>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>> available,
>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>>>> function
>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>> [4]PETSC ERROR: [4]
>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>> [4]PETSC ERROR: See
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>> shooting.
>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by
>>>>>>> vef002 Fri Jan 11 09:13:23 2019
>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>
>>>>>>> --------------------------------------------------------------------------
>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>> with errorcode 59.
>>>>>>>
>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>>> You may or may not see output from other processes, depending on
>>>>>>> exactly when Open MPI kills them.
>>>>>>>
>>>>>>> --------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>> the batch system) has told this process to end
>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>> -on_error_attach_debugger
>>>>>>> [0]PETSC ERROR: or see
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>
>>>>>>> Using Valgrind on only one of the valgrind files the following error
>>>>>>> was written:
>>>>>>>
>>>>>>> ==9053== Invalid read of size 4
>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>> (matmatmult.c:790)
>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>> (mpimatmatmult.c:1186)
>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently)
>>>>>>> free'd
>>>>>>> ==9053==
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>>>
>>>>>>>> Thank you Dave,
>>>>>>>>
>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the
>>>>>>>> code again with the following options:
>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>> gamg -log_view
>>>>>>>> (as on the petsc website you linked)
>>>>>>>>
>>>>>>>> It finished solving using the iterative solver, but the resulting
>>>>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This likely implies that you have a memory error in your code (a
>>>>>>>>> memory leak would not cause this behaviour).
>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>> errors.
>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>
>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>
>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>
>>>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>>>
>>>>>>>>>> how I run it in the job script on 2 node with 32 processors using
>>>>>>>>>> the clusters OpenMPI.
>>>>>>>>>>
>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>
>>>>>>>>>> the matrix:
>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>
>>>>>>>>>> Thanks and all the best
>>>>>>>>>>
>>>>>>>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/f9fcc7c7/attachment-0001.html>
-------------- next part --------------
static char help[] = "Solves a complex sparse linear system in parallel with KSP.\n\n";

#include <petscksp.h>
#include <iostream>
int main(int argc,char **args){
    Vec            x,b;           /* approx solution, RHS */
    Mat            A, F;             /* linear system matrix */
    KSP            ksp;           /* linear solver context */
    // PetscReal      norm;          /* norm of solution error */
    PC             pc;            /* petsc preconditioner object */
    PetscMPIInt    rank, size;    /* petsc mpi rank and size object */
    PetscViewer    viewer;        /* petsc viewer object needed for vis. and reading */
    PetscErrorCode ierr;

    // Initialise PETSc needed in the beginning of program, includes MPI initialisation. 
    ierr = PetscInitialize(&argc,&args,(char*)0,help);if (ierr) return ierr;
    MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
    MPI_Comm_size(PETSC_COMM_WORLD,&size);

    // Check if configured with complex
    #if !defined(PETSC_USE_COMPLEX)
      SETERRQ(PETSC_COMM_WORLD,1,"This example requires complex numbers");
    #endif
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
           Read the matrix and right-hand-side vector that defines
           the linear system, Ax = b.
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Reading vector in binary from Vector_b.dat ...\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"../petscpy/petscFiles/Vector_b.dat",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
    ierr = VecCreate(PETSC_COMM_WORLD, &b);CHKERRQ(ierr);
    ierr = VecLoad(b,viewer); CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Vector b read.\n");CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

    ierr = PetscPrintf(PETSC_COMM_WORLD,"Reading matrix in binary from Matrix_A.dat ...\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"../petscpy/petscFiles/Matrix_A.dat",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
    ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
    ierr = MatLoad(A,viewer);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Matrix A read.\n");CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
    // Duplicate vector b for solution x, content not duplicated.
    ierr = VecDuplicate(b,&x);CHKERRQ(ierr);
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                  Create the linear solver and set various options
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

    
    // Create linear solver context
    ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);

    /* Set operators. Here the matrix that defines the linear system
       also serves as the preconditioning constructor matrix.*/
    ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr);

    /* Set runtime options, e.g., -ksp_type <type> -pc_type <type>
     -ksp_monitor -ksp_rtol <rtol>. Here MUMPS is used if no runtime options is provided. */
    // ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr);
    // ierr = KSPGetPC(ksp, &pc);CHKERRQ(ierr);
    // ierr = PCSetType(pc, PCLU);CHKERRQ(ierr);
    // ierr = PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);CHKERRQ(ierr);
    ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);

    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                        Solve the linear system
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    /* KSPSetUp and KSPSetUpOnBlocks are optional. 
    Used to enable more precise profiling of setting up the preconditioner */ 
    ierr = KSPSetUp(ksp);CHKERRQ(ierr);
    // ierr = PCFactorGetMatrix(pc, &F);
    ierr = KSPSetUpOnBlocks(ksp);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Solving the system Ax=b\n");CHKERRQ(ierr);
    ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Done solving.\n");CHKERRQ(ierr);

    // Uncomment the following to view the solution in terminal
    // ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                        Save solution and clean up
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    ierr = PetscPrintf(PETSC_COMM_WORLD, "Writing solution x in PETSc binary file Vector_x_petsc.dat\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "../petscpy/petscFiles/Vector_x_petsc.dat", FILE_MODE_WRITE, &viewer);CHKERRQ(ierr);
    ierr = PetscViewerBinarySetSkipHeader(viewer, PETSC_TRUE);CHKERRQ(ierr);
    ierr = VecView(x,viewer);CHKERRQ(ierr);CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
    /*
       Free work space.  All PETSc objects should be destroyed when they
       are no longer needed.
    */
    ierr = KSPDestroy(&ksp);CHKERRQ(ierr);
    ierr = VecDestroy(&x);CHKERRQ(ierr);
    ierr = VecDestroy(&b);CHKERRQ(ierr); 
    ierr = MatDestroy(&A);CHKERRQ(ierr);
    ierr = PetscFinalize();
    return ierr;
}


/*TEST

   build:
      requires: complex, MUMPS
      optional: SuperLU_dist, SuperLU

   test:
      mpiexec -n 4 ./test -ksp_type gmres -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual

TEST*/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OneStripAntenna.e41273
Type: application/octet-stream
Size: 21108 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/f9fcc7c7/attachment-0001.obj>

From knepley at gmail.com  Thu Jan 17 07:59:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 08:59:16 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
Message-ID: <CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>

On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:

> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>
> I will try that, although I think solving the real problem (later on if I
> can get this to work) with 30 million finite elements might be a problem
> for SuperLU_dist. so it is better to get an iterative solver to work with
> first.
>
> 1) Try using    -build_twosided allreduce    on this run
>>
> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
> -ksp_monitor -log_view
> I have attached the full error output.
>

You are getting an SEGV on MatSetValues(), so its either

1) Running out of memory

2) You passed an invalid array


> 2) Is it possible to get something that fails here but we can run. None of
>> our tests show this problem.
>>
> I am not how I can do that, but i have added my code which is quite short
> and should only read and solve the system, the problem arises at larger
> matrices for example current test case has 6 million finite elements (~2B
> non-zero numbers and 25M x 25M matrix).
>

Are you running with 64-bit ints here?

   Matt


> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> The memory requested is an insane number. You may need to use 64 bit
>>>> integers.
>>>
>>> Thanks Mark, I reconfigured it to use 64bit, however in the process it
>>> says I can no longer use MUMPS and SuperLU as they are not supported (I see
>>> on MUMPS webpage it supports 64int). However it does not exactly solve the
>>> problem.
>>>
>>
>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>
>>
>>> This time, it crashes at
>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line
>>>> 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>> which allocates local portion of B^T*A.
>>>> You may also try to increase number of cores to reduce local matrix
>>>> size.
>>>>
>>>
>>> So I increased the number of cores to 16 on one node and ran it by :
>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>> It crashed after reading in the matrix and before starting to solve. The
>>> error:
>>>
>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [2]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [3]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [4]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [8]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [12]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>>
>>> --------------------------------------------------------------------------
>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on
>>> signal 9 (Killed).
>>>
>>> Now I was running this with valgrind as someone had previously suggested
>>> and the 16 files created all contain the same type of error:
>>>
>>
>> Okay, its possible that there are bugs in the MPI implementation. So
>>
>> 1) Try using    -build_twosided allreduce    on this run
>>
>> 2) Is it possible to get something that fails here but we can run. None
>> of our tests show this problem.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> ==25940== Invalid read of size 8
>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>> (mpits.c:373)
>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>> alloc'd
>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>> (mpits.c:371)
>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>> ==25940==
>>>
>>>
>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>>>
>>>> Fande:
>>>>
>>>>> According to this PR
>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>
>>>>> Should we set the scalable algorithm as default?
>>>>>
>>>> Sure, we can. But I feel we need do more tests to compare scalable and
>>>> non-scalable algorithms.
>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>> algorithm enables more efficient
>>>> data accessing. Andreas optimized scalable implementation. Our
>>>> non-scalable implementation might have room to be further optimized.
>>>> Hong
>>>>
>>>>>
>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>> Hong
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> I saw the following error message in your first email.
>>>>>>>
>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>
>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>
>>>>>>> --Junchao Zhang
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> Using a larger problem set with 2B non-zero elements and a matrix
>>>>>>>> of 25M x 25M I get the following error:
>>>>>>>> [4]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>> Violation, probably memory access out of range
>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>>> Mac OS X to find memory corruption errors
>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>> ------------------------------------
>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>>> available,
>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>>>>> function
>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>> [4]PETSC ERROR: See
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>> shooting.
>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03
>>>>>>>> by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>> with errorcode 59.
>>>>>>>>
>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>>> the batch system) has told this process to end
>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>> -on_error_attach_debugger
>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>
>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>> error was written:
>>>>>>>>
>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>> (matmatmult.c:790)
>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>> (recently) free'd
>>>>>>>> ==9053==
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you Dave,
>>>>>>>>>
>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the
>>>>>>>>> code again with the following options:
>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>> gamg -log_view
>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>
>>>>>>>>> It finished solving using the iterative solver, but the resulting
>>>>>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This likely implies that you have a memory error in your code (a
>>>>>>>>>> memory leak would not cause this behaviour).
>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>> errors.
>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>
>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>
>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>
>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>>>>
>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>
>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>
>>>>>>>>>>> the matrix:
>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>
>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>
>>>>>>>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/20e8f61e/attachment-0001.html>

From tempohoper at gmail.com  Thu Jan 17 08:18:30 2019
From: tempohoper at gmail.com (Sal Am)
Date: Thu, 17 Jan 2019 14:18:30 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
Message-ID: <CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>

>
> 1) Running out of memory
>
> 2) You passed an invalid array
>
I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e.
using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors
per node (128 processors in total).
I am not sure what would constitute an invalid array or how I can check
that. I am using the same procedure as when dealing with the smaller
matrix. i.e. Generate matrix A and vector b using FEM software then convert
the matrix and vector using a python script ready for petsc. read in petsc
and calculate.

Are you running with 64-bit ints here?
>
Yes I have it configured petsc with  --with-64-bit-indices and debugging
mode, which this was run on.

On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>
>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>
>> I will try that, although I think solving the real problem (later on if I
>> can get this to work) with 30 million finite elements might be a problem
>> for SuperLU_dist. so it is better to get an iterative solver to work with
>> first.
>>
>> 1) Try using    -build_twosided allreduce    on this run
>>>
>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>> -ksp_monitor -log_view
>> I have attached the full error output.
>>
>
> You are getting an SEGV on MatSetValues(), so its either
>
> 1) Running out of memory
>
> 2) You passed an invalid array
>
>
>> 2) Is it possible to get something that fails here but we can run. None
>>> of our tests show this problem.
>>>
>> I am not how I can do that, but i have added my code which is quite short
>> and should only read and solve the system, the problem arises at larger
>> matrices for example current test case has 6 million finite elements (~2B
>> non-zero numbers and 25M x 25M matrix).
>>
>
> Are you running with 64-bit ints here?
>
>    Matt
>
>
>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> The memory requested is an insane number. You may need to use 64 bit
>>>>> integers.
>>>>
>>>> Thanks Mark, I reconfigured it to use 64bit, however in the process it
>>>> says I can no longer use MUMPS and SuperLU as they are not supported (I see
>>>> on MUMPS webpage it supports 64int). However it does not exactly solve the
>>>> problem.
>>>>
>>>
>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>
>>>
>>>> This time, it crashes at
>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line
>>>>> 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>> which allocates local portion of B^T*A.
>>>>> You may also try to increase number of cores to reduce local matrix
>>>>> size.
>>>>>
>>>>
>>>> So I increased the number of cores to 16 on one node and ran it by :
>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>> It crashed after reading in the matrix and before starting to solve.
>>>> The error:
>>>>
>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>>> batch system) has told this process to end
>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [2]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>>> batch system) has told this process to end
>>>> [3]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [4]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [8]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [12]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>> the batch system) has told this process to end
>>>>
>>>> --------------------------------------------------------------------------
>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited
>>>> on signal 9 (Killed).
>>>>
>>>> Now I was running this with valgrind as someone had previously
>>>> suggested and the 16 files created all contain the same type of error:
>>>>
>>>
>>> Okay, its possible that there are bugs in the MPI implementation. So
>>>
>>> 1) Try using    -build_twosided allreduce    on this run
>>>
>>> 2) Is it possible to get something that fails here but we can run. None
>>> of our tests show this problem.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> ==25940== Invalid read of size 8
>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>> (mpits.c:373)
>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>>> alloc'd
>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>> (mpits.c:371)
>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>> ==25940==
>>>>
>>>>
>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>>>>
>>>>> Fande:
>>>>>
>>>>>> According to this PR
>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>
>>>>>> Should we set the scalable algorithm as default?
>>>>>>
>>>>> Sure, we can. But I feel we need do more tests to compare scalable and
>>>>> non-scalable algorithms.
>>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>>> algorithm enables more efficient
>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>> non-scalable implementation might have room to be further optimized.
>>>>> Hong
>>>>>
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>> Hong
>>>>>>>
>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> I saw the following error message in your first email.
>>>>>>>>
>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>
>>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>
>>>>>>>> --Junchao Zhang
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Using a larger problem set with 2B non-zero elements and a matrix
>>>>>>>>> of 25M x 25M I get the following error:
>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>>>>>>> Mac OS X to find memory corruption errors
>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>> ------------------------------------
>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>>>> available,
>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>>>>>> function
>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>> shooting.
>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03
>>>>>>>>> by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>>> with errorcode 59.
>>>>>>>>>
>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>> -on_error_attach_debugger
>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>
>>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>>> error was written:
>>>>>>>>>
>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>> (matmatmult.c:790)
>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>> (recently) free'd
>>>>>>>>> ==9053==
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you Dave,
>>>>>>>>>>
>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the
>>>>>>>>>> code again with the following options:
>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>> gamg -log_view
>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>
>>>>>>>>>> It finished solving using the iterative solver, but the resulting
>>>>>>>>>> valgrind.log.%p files (all 8 corresponding to each processor) are all
>>>>>>>>>> empty. And it took a whooping ~15hours, for what used to take ~10-20min.
>>>>>>>>>> Maybe this is because of valgrind? I am not sure. Attached is the log_view.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This likely implies that you have a memory error in your code (a
>>>>>>>>>>> memory leak would not cause this behaviour).
>>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>>> errors.
>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>
>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>
>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>
>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>>>>>
>>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>>
>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>
>>>>>>>>>>>> the matrix:
>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>
>>>>>>>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/07bf2421/attachment-0001.html>

From knepley at gmail.com  Thu Jan 17 10:24:09 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 11:24:09 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
Message-ID: <CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>

On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:

> 1) Running out of memory
>>
>> 2) You passed an invalid array
>>
> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e.
> using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors
> per node (128 processors in total).
> I am not sure what would constitute an invalid array or how I can check
> that. I am using the same procedure as when dealing with the smaller
> matrix. i.e. Generate matrix A and vector b using FEM software then convert
> the matrix and vector using a python script ready for petsc. read in petsc
> and calculate.
>
> Are you running with 64-bit ints here?
>>
> Yes I have it configured petsc with  --with-64-bit-indices and debugging
> mode, which this was run on.
>

It sounds like you have enough memory, but the fact that is runs for
smaller problems makes me suspicious. It
could still be a memory overwrite. Can you either

a) Run under valgrind

or

b) Run under the debugger and get a stack trace

 ?

   Thanks,

    Matt


> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>>
>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>
>>> I will try that, although I think solving the real problem (later on if
>>> I can get this to work) with 30 million finite elements might be a problem
>>> for SuperLU_dist. so it is better to get an iterative solver to work with
>>> first.
>>>
>>> 1) Try using    -build_twosided allreduce    on this run
>>>>
>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>> -ksp_monitor -log_view
>>> I have attached the full error output.
>>>
>>
>> You are getting an SEGV on MatSetValues(), so its either
>>
>> 1) Running out of memory
>>
>> 2) You passed an invalid array
>>
>>
>>> 2) Is it possible to get something that fails here but we can run. None
>>>> of our tests show this problem.
>>>>
>>> I am not how I can do that, but i have added my code which is quite
>>> short and should only read and solve the system, the problem arises at
>>> larger matrices for example current test case has 6 million finite elements
>>> (~2B non-zero numbers and 25M x 25M matrix).
>>>
>>
>> Are you running with 64-bit ints here?
>>
>>    Matt
>>
>>
>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> The memory requested is an insane number. You may need to use 64 bit
>>>>>> integers.
>>>>>
>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the process it
>>>>> says I can no longer use MUMPS and SuperLU as they are not supported (I see
>>>>> on MUMPS webpage it supports 64int). However it does not exactly solve the
>>>>> problem.
>>>>>
>>>>
>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>
>>>>
>>>>> This time, it crashes at
>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line
>>>>>> 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>> which allocates local portion of B^T*A.
>>>>>> You may also try to increase number of cores to reduce local matrix
>>>>>> size.
>>>>>>
>>>>>
>>>>> So I increased the number of cores to 16 on one node and ran it by :
>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>> It crashed after reading in the matrix and before starting to solve.
>>>>> The error:
>>>>>
>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>> the batch system) has told this process to end
>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [2]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>> the batch system) has told this process to end
>>>>> [3]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [4]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [8]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [12]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>> the batch system) has told this process to end
>>>>>
>>>>> --------------------------------------------------------------------------
>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited
>>>>> on signal 9 (Killed).
>>>>>
>>>>> Now I was running this with valgrind as someone had previously
>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>
>>>>
>>>> Okay, its possible that there are bugs in the MPI implementation. So
>>>>
>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>
>>>> 2) Is it possible to get something that fails here but we can run. None
>>>> of our tests show this problem.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> ==25940== Invalid read of size 8
>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>> (mpits.c:373)
>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>>>> alloc'd
>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>> (mpits.c:371)
>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>> ==25940==
>>>>>
>>>>>
>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>> Fande:
>>>>>>
>>>>>>> According to this PR
>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>
>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>
>>>>>> Sure, we can. But I feel we need do more tests to compare scalable
>>>>>> and non-scalable algorithms.
>>>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>>>> algorithm enables more efficient
>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>> Hong
>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>> Hong
>>>>>>>>
>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>
>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>
>>>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>
>>>>>>>>> --Junchao Zhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a matrix
>>>>>>>>>> of 25M x 25M I get the following error:
>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>> ------------------------------------
>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>>>>>>> available,
>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>>>>>>> function
>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>>> shooting.
>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03
>>>>>>>>>> by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown
>>>>>>>>>> file
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>>>> with errorcode 59.
>>>>>>>>>>
>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>> processes.
>>>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>
>>>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>>>> error was written:
>>>>>>>>>>
>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>> (recently) free'd
>>>>>>>>>> ==9053==
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>
>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran the
>>>>>>>>>>> code again with the following options:
>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>> gamg -log_view
>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>
>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>> the log_view.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This likely implies that you have a memory error in your code
>>>>>>>>>>>> (a memory leak would not cause this behaviour).
>>>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>>>> errors.
>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>
>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>
>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it with
>>>>>>>>>>>>> -malloc_dump, but not getting anything printed out, however, when run with
>>>>>>>>>>>>> -log_view I see that Viewer is created 4 times, but destroyed 3 times. The
>>>>>>>>>>>>> way I see it, I have destroyed it where I see I no longer have use for it
>>>>>>>>>>>>> so not sure if I am wrong. Could this be the reason why it keeps crashing?
>>>>>>>>>>>>> It crashes as soon as it reads the matrix, before entering the solving mode
>>>>>>>>>>>>> (I have a print statement before solving starts that never prints).
>>>>>>>>>>>>>
>>>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>>>
>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>
>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/e79342ef/attachment-0001.html>

From tempohoper at gmail.com  Thu Jan 17 10:43:26 2019
From: tempohoper at gmail.com (Sal Am)
Date: Thu, 17 Jan 2019 16:43:26 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
Message-ID: <CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>

I did two runs, one with the SuperLU_dist and one with bcgs using jacobi,
attached are the results of one of the reports from valgrind on one random
processor (out of the 128 files).

DS = direct solver
IS = iterative  solver

There is an awful lot of errors.

how I initiated the two runs:
mpiexec valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs
-pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce
-ksp_monitor -log_view

mpiexec valgrind --tool=memcheck -q --num-callers=20
--log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
-pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
-ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

Thank you

On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:
>
>> 1) Running out of memory
>>>
>>> 2) You passed an invalid array
>>>
>> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e.
>> using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors
>> per node (128 processors in total).
>> I am not sure what would constitute an invalid array or how I can check
>> that. I am using the same procedure as when dealing with the smaller
>> matrix. i.e. Generate matrix A and vector b using FEM software then convert
>> the matrix and vector using a python script ready for petsc. read in petsc
>> and calculate.
>>
>> Are you running with 64-bit ints here?
>>>
>> Yes I have it configured petsc with  --with-64-bit-indices and debugging
>> mode, which this was run on.
>>
>
> It sounds like you have enough memory, but the fact that is runs for
> smaller problems makes me suspicious. It
> could still be a memory overwrite. Can you either
>
> a) Run under valgrind
>
> or
>
> b) Run under the debugger and get a stack trace
>
>  ?
>
>    Thanks,
>
>     Matt
>
>
>> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>>>
>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>
>>>> I will try that, although I think solving the real problem (later on if
>>>> I can get this to work) with 30 million finite elements might be a problem
>>>> for SuperLU_dist. so it is better to get an iterative solver to work with
>>>> first.
>>>>
>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>
>>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>>> -ksp_monitor -log_view
>>>> I have attached the full error output.
>>>>
>>>
>>> You are getting an SEGV on MatSetValues(), so its either
>>>
>>> 1) Running out of memory
>>>
>>> 2) You passed an invalid array
>>>
>>>
>>>> 2) Is it possible to get something that fails here but we can run. None
>>>>> of our tests show this problem.
>>>>>
>>>> I am not how I can do that, but i have added my code which is quite
>>>> short and should only read and solve the system, the problem arises at
>>>> larger matrices for example current test case has 6 million finite elements
>>>> (~2B non-zero numbers and 25M x 25M matrix).
>>>>
>>>
>>> Are you running with 64-bit ints here?
>>>
>>>    Matt
>>>
>>>
>>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> The memory requested is an insane number. You may need to use 64 bit
>>>>>>> integers.
>>>>>>
>>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the process
>>>>>> it says I can no longer use MUMPS and SuperLU as they are not supported (I
>>>>>> see on MUMPS webpage it supports 64int). However it does not exactly solve
>>>>>> the problem.
>>>>>>
>>>>>
>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>
>>>>>
>>>>>> This time, it crashes at
>>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line
>>>>>>> 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>>> which allocates local portion of B^T*A.
>>>>>>> You may also try to increase number of cores to reduce local matrix
>>>>>>> size.
>>>>>>>
>>>>>>
>>>>>> So I increased the number of cores to 16 on one node and ran it by :
>>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>>> It crashed after reading in the matrix and before starting to solve.
>>>>>> The error:
>>>>>>
>>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>> the batch system) has told this process to end
>>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [2]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>> the batch system) has told this process to end
>>>>>> [3]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [4]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [8]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [12]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>> the batch system) has told this process to end
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited
>>>>>> on signal 9 (Killed).
>>>>>>
>>>>>> Now I was running this with valgrind as someone had previously
>>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>>
>>>>>
>>>>> Okay, its possible that there are bugs in the MPI implementation. So
>>>>>
>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>
>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>> None of our tests show this problem.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> ==25940== Invalid read of size 8
>>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>> (mpits.c:373)
>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>>>>> alloc'd
>>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>> (mpits.c:371)
>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>> ==25940==
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>>> wrote:
>>>>>>
>>>>>>> Fande:
>>>>>>>
>>>>>>>> According to this PR
>>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>>
>>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>>
>>>>>>> Sure, we can. But I feel we need do more tests to compare scalable
>>>>>>> and non-scalable algorithms.
>>>>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>>>>> algorithm enables more efficient
>>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>>> Hong
>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>>> Hong
>>>>>>>>>
>>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>>
>>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>>
>>>>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>>
>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a
>>>>>>>>>>> matrix of 25M x 25M I get the following error:
>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>>> ------------------------------------
>>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>>>>> not available,
>>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of
>>>>>>>>>>> the function
>>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>>>>>>> shooting.
>>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named
>>>>>>>>>>> r02g03 by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>>> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown
>>>>>>>>>>> file
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>>>>> with errorcode 59.
>>>>>>>>>>>
>>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>>> processes.
>>>>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>
>>>>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>>>>> error was written:
>>>>>>>>>>>
>>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>>> (recently) free'd
>>>>>>>>>>> ==9053==
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>>
>>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran
>>>>>>>>>>>> the code again with the following options:
>>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>>> gamg -log_view
>>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>>
>>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>>> the log_view.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This likely implies that you have a memory error in your code
>>>>>>>>>>>>> (a memory leak would not cause this behaviour).
>>>>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>>>>> errors.
>>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>
>>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it
>>>>>>>>>>>>>> with -malloc_dump, but not getting anything printed out, however, when run
>>>>>>>>>>>>>> with -log_view I see that Viewer is created 4 times, but destroyed 3 times.
>>>>>>>>>>>>>> The way I see it, I have destroyed it where I see I no longer have use for
>>>>>>>>>>>>>> it so not sure if I am wrong. Could this be the reason why it keeps
>>>>>>>>>>>>>> crashing? It crashes as soon as it reads the matrix, before entering the
>>>>>>>>>>>>>> solving mode (I have a print statement before solving starts that never
>>>>>>>>>>>>>> prints).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/48c0c117/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind.log-DS.25719
Type: application/octet-stream
Size: 233920 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/48c0c117/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind.log-IS.56418
Type: application/octet-stream
Size: 318003 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/48c0c117/attachment-0003.obj>

From knepley at gmail.com  Thu Jan 17 11:37:54 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 12:37:54 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
Message-ID: <CAMYG4GmVt9JSWvapsn=iv-W1N_E+4jm5PCBFo=Kb1KpuGDprpA@mail.gmail.com>

On Thu, Jan 17, 2019 at 11:43 AM Sal Am <tempohoper at gmail.com> wrote:

> I did two runs, one with the SuperLU_dist and one with bcgs using jacobi,
> attached are the results of one of the reports from valgrind on one random
> processor (out of the 128 files).
>
> DS = direct solver
> IS = iterative  solver
>
> There is an awful lot of errors.
>
> how I initiated the two runs:
> mpiexec valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs
> -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce
> -ksp_monitor -log_view
>
> mpiexec valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>

OpenMPI is not my favorite. You need to use a suppressions file to get rid
of all of that noise. Here is one:


https://github.com/geodynamics/pylith/blob/master/share/valgrind-openmpi.supp

  Thanks,

    Matt


> Thank you
>
> On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:
>>
>>> 1) Running out of memory
>>>>
>>>> 2) You passed an invalid array
>>>>
>>> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e.
>>> using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors
>>> per node (128 processors in total).
>>> I am not sure what would constitute an invalid array or how I can check
>>> that. I am using the same procedure as when dealing with the smaller
>>> matrix. i.e. Generate matrix A and vector b using FEM software then convert
>>> the matrix and vector using a python script ready for petsc. read in petsc
>>> and calculate.
>>>
>>> Are you running with 64-bit ints here?
>>>>
>>> Yes I have it configured petsc with  --with-64-bit-indices and debugging
>>> mode, which this was run on.
>>>
>>
>> It sounds like you have enough memory, but the fact that is runs for
>> smaller problems makes me suspicious. It
>> could still be a memory overwrite. Can you either
>>
>> a) Run under valgrind
>>
>> or
>>
>> b) Run under the debugger and get a stack trace
>>
>>  ?
>>
>>    Thanks,
>>
>>     Matt
>>
>>
>>> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>
>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>
>>>>> I will try that, although I think solving the real problem (later on
>>>>> if I can get this to work) with 30 million finite elements might be a
>>>>> problem for SuperLU_dist. so it is better to get an iterative solver to
>>>>> work with first.
>>>>>
>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>
>>>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>>>> -ksp_monitor -log_view
>>>>> I have attached the full error output.
>>>>>
>>>>
>>>> You are getting an SEGV on MatSetValues(), so its either
>>>>
>>>> 1) Running out of memory
>>>>
>>>> 2) You passed an invalid array
>>>>
>>>>
>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>> None of our tests show this problem.
>>>>>>
>>>>> I am not how I can do that, but i have added my code which is quite
>>>>> short and should only read and solve the system, the problem arises at
>>>>> larger matrices for example current test case has 6 million finite elements
>>>>> (~2B non-zero numbers and 25M x 25M matrix).
>>>>>
>>>>
>>>> Are you running with 64-bit ints here?
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> The memory requested is an insane number. You may need to use 64 bit
>>>>>>>> integers.
>>>>>>>
>>>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the process
>>>>>>> it says I can no longer use MUMPS and SuperLU as they are not supported (I
>>>>>>> see on MUMPS webpage it supports 64int). However it does not exactly solve
>>>>>>> the problem.
>>>>>>>
>>>>>>
>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>
>>>>>>
>>>>>>> This time, it crashes at
>>>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line
>>>>>>>> 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>>>> which allocates local portion of B^T*A.
>>>>>>>> You may also try to increase number of cores to reduce local matrix
>>>>>>>> size.
>>>>>>>>
>>>>>>>
>>>>>>> So I increased the number of cores to 16 on one node and ran it by :
>>>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>>>> It crashed after reading in the matrix and before starting to solve.
>>>>>>> The error:
>>>>>>>
>>>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>> the batch system) has told this process to end
>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>> the batch system) has told this process to end
>>>>>>> [3]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [4]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [8]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [12]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>> the batch system) has told this process to end
>>>>>>>
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01
>>>>>>> exited on signal 9 (Killed).
>>>>>>>
>>>>>>> Now I was running this with valgrind as someone had previously
>>>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>>>
>>>>>>
>>>>>> Okay, its possible that there are bugs in the MPI implementation. So
>>>>>>
>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>
>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>> None of our tests show this problem.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> ==25940== Invalid read of size 8
>>>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>> (mpits.c:373)
>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>>>>>> alloc'd
>>>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>> (mpits.c:371)
>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>> ==25940==
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Fande:
>>>>>>>>
>>>>>>>>> According to this PR
>>>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>>>
>>>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>>>
>>>>>>>> Sure, we can. But I feel we need do more tests to compare scalable
>>>>>>>> and non-scalable algorithms.
>>>>>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>>>>>> algorithm enables more efficient
>>>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>>>> Hong
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>>>> Hong
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>>>
>>>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>>>
>>>>>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>>>
>>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a
>>>>>>>>>>>> matrix of 25M x 25M I get the following error:
>>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>>>>>> not available,
>>>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of
>>>>>>>>>>>> the function
>>>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>>>>>>>>>>> trouble shooting.
>>>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named
>>>>>>>>>>>> r02g03 by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>>>> [4]PETSC ERROR: Configure options
>>>>>>>>>>>> PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown
>>>>>>>>>>>> file
>>>>>>>>>>>>
>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>>>>>> with errorcode 59.
>>>>>>>>>>>>
>>>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>>>> processes.
>>>>>>>>>>>> You may or may not see output from other processes, depending on
>>>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>>>
>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>
>>>>>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>>>>>> error was written:
>>>>>>>>>>>>
>>>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
>>>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>>>> (recently) free'd
>>>>>>>>>>>> ==9053==
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran
>>>>>>>>>>>>> the code again with the following options:
>>>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>>>> gamg -log_view
>>>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>>>
>>>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>>>> the log_view.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This likely implies that you have a memory error in your code
>>>>>>>>>>>>>> (a memory leak would not cause this behaviour).
>>>>>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it
>>>>>>>>>>>>>>> with -malloc_dump, but not getting anything printed out, however, when run
>>>>>>>>>>>>>>> with -log_view I see that Viewer is created 4 times, but destroyed 3 times.
>>>>>>>>>>>>>>> The way I see it, I have destroyed it where I see I no longer have use for
>>>>>>>>>>>>>>> it so not sure if I am wrong. Could this be the reason why it keeps
>>>>>>>>>>>>>>> crashing? It crashes as soon as it reads the matrix, before entering the
>>>>>>>>>>>>>>> solving mode (I have a print statement before solving starts that never
>>>>>>>>>>>>>>> prints).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/4305a805/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Thu Jan 17 13:18:12 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 17 Jan 2019 13:18:12 -0600
Subject: [petsc-users] VecCopy fails after VecDuplicate
Message-ID: <CAOGsD9hQGFJ+xyVk36-90CE9rZ3bEVo4qWcTe2z6DuMqHdAZEA@mail.gmail.com>

Hi,

I have the following 2 lines in a function in my code :

 ierr = VecDuplicate(u,&u_abs);CHKERRQ(ierr);
 ierr = VecCopy(u,u_abs);CHKERRQ(ierr);

The VecCopy fails with the error message :
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled vector

adding the following statements doesn't help either :
ierr = VecAssemblyBegin(u_abs);CHKERRQ(ierr);
ierr = VecAssemblyEnd(u_abs);CHKERRQ(ierr);

If needed, the entire file is at :
https://github.com/s-sajid-ali/xwp_petsc/blob/master/1d/free_space/ex_modify.c

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/808bd2ea/attachment.html>

From bsmith at mcs.anl.gov  Thu Jan 17 13:26:41 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 17 Jan 2019 19:26:41 +0000
Subject: [petsc-users] VecCopy fails after VecDuplicate
In-Reply-To: <CAOGsD9hQGFJ+xyVk36-90CE9rZ3bEVo4qWcTe2z6DuMqHdAZEA@mail.gmail.com>
References: <CAOGsD9hQGFJ+xyVk36-90CE9rZ3bEVo4qWcTe2z6DuMqHdAZEA@mail.gmail.com>
Message-ID: <B9616EDB-7725-47A2-BEDF-55C192587ADE@anl.gov>


  It is likely the u vector that is not assembled not the u_abs; the complete error message has this information. If the monitor is called from TS then I cannot imagine how the u could possibly be unassembled. Please send the complete error message.

  Barry


> On Jan 17, 2019, at 1:18 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi, 
> 
> I have the following 2 lines in a function in my code :
> 
>  ierr = VecDuplicate(u,&u_abs);CHKERRQ(ierr);
>  ierr = VecCopy(u,u_abs);CHKERRQ(ierr);
> 
> The VecCopy fails with the error message : 
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled vector
> 
> adding the following statements doesn't help either :
> ierr = VecAssemblyBegin(u_abs);CHKERRQ(ierr);
> ierr = VecAssemblyEnd(u_abs);CHKERRQ(ierr);
> 
> If needed, the entire file is at : https://github.com/s-sajid-ali/xwp_petsc/blob/master/1d/free_space/ex_modify.c
> 
> Thank You, 
> Sajid Ali
> Applied Physics
> Northwestern University


From sajidsyed2021 at u.northwestern.edu  Thu Jan 17 13:29:37 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 17 Jan 2019 13:29:37 -0600
Subject: [petsc-users] VecCopy fails after VecDuplicate
In-Reply-To: <B9616EDB-7725-47A2-BEDF-55C192587ADE@anl.gov>
References: <CAOGsD9hQGFJ+xyVk36-90CE9rZ3bEVo4qWcTe2z6DuMqHdAZEA@mail.gmail.com>
	<B9616EDB-7725-47A2-BEDF-55C192587ADE@anl.gov>
Message-ID: <CAOGsD9iDzLL+jokQdsyZUG+jOBLFkW4rTCeW6uTBznSrpXqqiw@mail.gmail.com>

As requested :

[sajid at xrm free_space]$ ./ex_modify
Solving a linear TS problem on 1 processor
m : 256, slices : 1000.000000, lambda : 1.239800e-10
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled vector
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision:
0cd88d33dca7e1f18a10cbb6fcb08f83d068c5f4  GIT Date: 2019-01-06 13:27:26
-0600
[0]PETSC ERROR: ./ex_modify on a  named xrm by sajid Thu Jan 17 13:29:12
2019
[0]PETSC ERROR: Configure options
--prefix=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj
--with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0
CFLAGS= FFLAGS= CXXFLAGS=
--with-cc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc
--with-cxx=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpic++
--with-fc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpif90
--with-precision=double --with-scalar-type=complex
--with-shared-libraries=1 --with-debugging=1 --with-64-bit-indices=0
--with-debugging=%s
--with-blaslapack-lib="/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_intel_lp64.so
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_sequential.so
/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_core.so
/lib64/libpthread.so /lib64/libm.so /lib64/libdl.so" --with-x=1
--with-clanguage=C --with-scalapack=0 --with-metis=1
--with-metis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/metis-5.1.0-nhgzn4kjskctzmzv35mstvd34nj2ugek
--with-hdf5=1
--with-hdf5-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww
--with-hypre=0 --with-parmetis=1
--with-parmetis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/parmetis-4.0.3-hw3j2ss7mjsc5x5f2gaflirnuufzptil
--with-mumps=0 --with-trilinos=0 --with-cxx-dialect=C++11
--with-superlu_dist-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/include
--with-superlu_dist-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/lib/libsuperlu_dist.a
--with-superlu_dist=1 --with-suitesparse=0
--with-zlib-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/include
--with-zlib-lib="-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/lib
-lz" --with-zlib=1
[0]PETSC ERROR: #1 VecCopy() line 1571 in
/tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #2 Monitor() line 296 in
/raid/home/sajid/packages/xwp_petsc/1d/free_space/ex_modify.c
[0]PETSC ERROR: #3 TSMonitor() line 3929 in
/tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #4 TSSolve() line 3843 in
/tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #5 main() line 188 in
/raid/home/sajid/packages/xwp_petsc/1d/free_space/ex_modify.c
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=73
:
system msg for write_line failure : Bad file descriptor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/3c5564b7/attachment.html>

From bsmith at mcs.anl.gov  Thu Jan 17 13:32:30 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 17 Jan 2019 19:32:30 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
Message-ID: <FB47E2FA-6C19-4405-A75A-8E89D15B90F6@anl.gov>

0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end

   This happens because you have run out of time requested on your batch system so the batch system is terminating the processes or
you are running low on memory (or some other resource) and so the operating system is terminating the processes. Perhaps the values 
for ulimit are not set high enough for your processes to utilize all the memory? Or you are getting close to the physical memory limits.


   Barry


> On Jan 17, 2019, at 7:16 AM, Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
> I will try that, although I think solving the real problem (later on if I can get this to work) with 30 million finite elements might be a problem for SuperLU_dist. so it is better to get an iterative solver to work with first.
> 
> 1) Try using    -build_twosided allreduce    on this run
> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce -ksp_monitor -log_view
> I have attached the full error output.
> 
> 2) Is it possible to get something that fails here but we can run. None of our tests show this problem.
> I am not how I can do that, but i have added my code which is quite short and should only read and solve the system, the problem arises at larger matrices for example current test case has 6 million finite elements (~2B non-zero numbers and 25M x 25M matrix). 
> 
> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> The memory requested is an insane number. You may need to use 64 bit integers.
> Thanks Mark, I reconfigured it to use 64bit, however in the process it says I can no longer use MUMPS and SuperLU as they are not supported (I see on MUMPS webpage it supports 64int). However it does not exactly solve the problem.
> 
> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>  
> This time, it crashes at 
> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> ierr      = PetscMalloc1(bi[pn]+1,&bj);
> which allocates local portion of B^T*A. 
> You may also try to increase number of cores to reduce local matrix size.
> 
> So I increased the number of cores to 16 on one node and ran it by :
> mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view 
> It crashed after reading in the matrix and before starting to solve. The error: 
> 
> [15]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR: ------------------------------------------------------------------------
> [8]PETSC ERROR: ------------------------------------------------------------------------
> [12]PETSC ERROR: ------------------------------------------------------------------------
> [12]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------
> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> --------------------------------------------------------------------------
> mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on signal 9 (Killed).
> 
> Now I was running this with valgrind as someone had previously suggested and the 16 files created all contain the same type of error:
> 
> Okay, its possible that there are bugs in the MPI implementation. So
> 
> 1) Try using    -build_twosided allreduce    on this run
> 
> 2) Is it possible to get something that fails here but we can run. None of our tests show this problem.
> 
>   Thanks,
> 
>      Matt
>  
> ==25940== Invalid read of size 8
> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:373)
> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16 alloc'd
> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:371)
> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
> ==25940==
> 
> 
> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> Fande:
> According to this PR https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
> 
> Should we set the scalable algorithm as default?
> Sure, we can. But I feel we need do more tests to compare scalable and non-scalable algorithms. 
> On theory, for small to medium matrices, non-scalable matmatmult() algorithm enables more efficient 
> data accessing. Andreas optimized scalable implementation. Our non-scalable implementation might have room to be further optimized. 
> Hong
> 
> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Add option '-mattransposematmult_via scalable'
> Hong
> 
> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov> wrote:
> I saw the following error message in your first email.
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.
> 
> --Junchao Zhang
> 
> 
> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [4]PETSC ERROR: likely location of problem given in stack below
> [4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [4]PETSC ERROR:       INSTEAD the line number of the start of the function
> [4]PETSC ERROR:       is given.
> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
> [4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
> [4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
> [4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
> [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [4]PETSC ERROR: Signal received
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown 
> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
> [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
> with errorcode 59.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> 
> Using Valgrind on only one of the valgrind files the following error was written:
> 
> ==9053== Invalid read of size 4
> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
> ==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
> ==9053==
> 
> 
> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com> wrote:
> Thank you Dave,
> 
> I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
> (as on the petsc website you linked)
> 
> It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.
> 
> 
> On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com> wrote:
> 
> 
> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).
> 
> This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
> I strongly suggest you make sure your code is free of memory errors.
> You can do this using valgrind. See here 
> 
> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> 
> for an explanation of how to use valgrind.
>  
> I have attached the first two run's errors and my code. 
> 
> Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).
> 
> how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI. 
> 
> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
> 
> the matrix:
> 2 122 821 366 (non-zero elements)
> 25 947 279 x 25 947 279
> 
> Thanks and all the best
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <solveCmplxLinearSys.cpp><OneStripAntenna.e41273>


From sajidsyed2021 at u.northwestern.edu  Thu Jan 17 13:34:08 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 17 Jan 2019 13:34:08 -0600
Subject: [petsc-users] VecCopy fails after VecDuplicate
In-Reply-To: <CAOGsD9iDzLL+jokQdsyZUG+jOBLFkW4rTCeW6uTBznSrpXqqiw@mail.gmail.com>
References: <CAOGsD9hQGFJ+xyVk36-90CE9rZ3bEVo4qWcTe2z6DuMqHdAZEA@mail.gmail.com>
	<B9616EDB-7725-47A2-BEDF-55C192587ADE@anl.gov>
	<CAOGsD9iDzLL+jokQdsyZUG+jOBLFkW4rTCeW6uTBznSrpXqqiw@mail.gmail.com>
Message-ID: <CAOGsD9i-7VwTRULHNwihE7w6dBhQjHkY4RBCaPFp+dKE8+WGCg@mail.gmail.com>

Nevermind, it was my fault in thinking the error was with u_abs and not u.
I switched from local array based value setting for initial conditions to
VecSetValues when converting the uniprocessor example to an MPI program.
While I removed VecRestoreArray and swapped u_local[*ptr] assignments with
VecSetValues, I missed out on adding VecAssembleBegin/End to compensate.
Thanks for pointing out that the error was with u and not u_abs.

On Thu, Jan 17, 2019 at 1:29 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> As requested :
>
> [sajid at xrm free_space]$ ./ex_modify
> Solving a linear TS problem on 1 processor
> m : 256, slices : 1000.000000, lambda : 1.239800e-10
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled vector
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision:
> 0cd88d33dca7e1f18a10cbb6fcb08f83d068c5f4  GIT Date: 2019-01-06 13:27:26
> -0600
> [0]PETSC ERROR: ./ex_modify on a  named xrm by sajid Thu Jan 17 13:29:12
> 2019
> [0]PETSC ERROR: Configure options
> --prefix=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-develop-2u6vuwagkoczyvnpsubzrubmtmpfhhkj
> --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0
> CFLAGS= FFLAGS= CXXFLAGS=
> --with-cc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpicc
> --with-cxx=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpic++
> --with-fc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/bin/mpif90
> --with-precision=double --with-scalar-type=complex
> --with-shared-libraries=1 --with-debugging=1 --with-64-bit-indices=0
> --with-debugging=%s
> --with-blaslapack-lib="/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_intel_lp64.so
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_sequential.so
> /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64/libmkl_core.so
> /lib64/libpthread.so /lib64/libm.so /lib64/libdl.so" --with-x=1
> --with-clanguage=C --with-scalapack=0 --with-metis=1
> --with-metis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/metis-5.1.0-nhgzn4kjskctzmzv35mstvd34nj2ugek
> --with-hdf5=1
> --with-hdf5-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww
> --with-hypre=0 --with-parmetis=1
> --with-parmetis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/parmetis-4.0.3-hw3j2ss7mjsc5x5f2gaflirnuufzptil
> --with-mumps=0 --with-trilinos=0 --with-cxx-dialect=C++11
> --with-superlu_dist-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/include
> --with-superlu_dist-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-develop-cpspq4ca2hnyvhx4mz7zsupbj3do6md3/lib/libsuperlu_dist.a
> --with-superlu_dist=1 --with-suitesparse=0
> --with-zlib-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/include
> --with-zlib-lib="-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/lib
> -lz" --with-zlib=1
> [0]PETSC ERROR: #1 VecCopy() line 1571 in
> /tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #2 Monitor() line 296 in
> /raid/home/sajid/packages/xwp_petsc/1d/free_space/ex_modify.c
> [0]PETSC ERROR: #3 TSMonitor() line 3929 in
> /tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #4 TSSolve() line 3843 in
> /tmp/sajid/spack-stage/spack-stage-nwxY3Q/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #5 main() line 188 in
> /raid/home/sajid/packages/xwp_petsc/1d/free_space/ex_modify.c
> [0]PETSC ERROR: No PETSc Option Table entries
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=73
> :
> system msg for write_line failure : Bad file descriptor
>
>

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/935a4139/attachment.html>

From leejearl at mail.nwpu.edu.cn  Thu Jan 17 20:04:24 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Fri, 18 Jan 2019 10:04:24 +0800
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
Message-ID: <d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>

Hi Matt,	Thanks for your reply.  We have got to a consensus. For
such a problem, I have sought the mail list for helps and found a great
many misleading answers. I think some of answers may work for the DMDA.
Now I have overcome this by creating a PetscSF and PetscSFBcast, and
the procedures are described as follow.

1. Using DMGLobaltoLocal to update the local vector(the ghost cell
mighb get the wrong result)2. Creating a PetscSF which matches the
local ghost cell and the global donor cell across the processes3. Using
PetscSFBCast to update the local ghost cell
Now, my code can work. Can you give me a more scalable procedure?
Thanks.
On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
> On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi all Petscer,
> > 
> >         I have ask helps for some questions. Thanks for the replies
> > 
> > from developer. I have known more about the petsc. I have also
> > sought
> > 
> > helps in the mail lists, and I find that there are many subjects
> > 
> > focused on such a problem. Some subjects are listed as follow
> > 
> > 
> > 
> > 
> > 
> > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> > 
> > ,
> > 
> > 
> > 
> > 
> > 
> > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> > 
> > ,
> > 
> > 
> > 
> > 
> > 
> > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> > 
> > ,
> > 
> > 
> > 
> > 
> > 
> > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
> > 
> > 
> > 
> > ......
> > 
> > 
> > 
> > The problem can be summarized as
> > 
> > 1. Getting the value of vector from other processor
> > 
> > 2. How to set the value of ghost cell, the value of the ghost cell
> > 
> > might be as same as a very cell in other processor. 
> > 
> > 
> > 
> > I think the above problem is very popular. I suffer such a problem
> > when
> > 
> > I want to treat the periodic boundary for the FVM method.
> 
> Ah, I think I am now seeing the problem. In DMPlex, we do not
> implementperiodicity of the mesh by putting in extra communication.
> We make aperiodic mesh directly. For example, in 1D what you describe
> would mesha line segment, and then communicate the values from one
> end to the other.Instead, in Plex we just mesh a circle.
>   Thanks,
>     Matt 
> >  After the dm
> > 
> > object is distributed, the donor cell of bound might be in other
> > 
> > processor. Since the donor cell must be matched as a pair
> > correctly,
> > 
> > using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get
> > the
> > 
> > expected results. 
> > 
> > 
> > 
> > In fact, such a problem is not very difficult for a mpi program.
> > But we
> > 
> > are coding with the petsc, we always think that whether we can
> > 
> > implement more easily.
> > 
> > 
> > 
> > Whether our developer can create a demo for us to following. I
> > think
> > 
> > such a demo is very useful for user.
> > 
> > 
> > 
> > Thanks
> > 
> > 
> > 
> > leejeal
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/5ec2a2dc/attachment.html>

From knepley at gmail.com  Thu Jan 17 20:20:09 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 21:20:09 -0500
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
	<d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>

On Thu, Jan 17, 2019 at 9:04 PM leejearl <leejearl at mail.nwpu.edu.cn> wrote:

> Hi Matt,
> Thanks for your reply. We have got to a consensus. For such a problem, I
> have sought the mail list for helps and found a great
> many misleading answers. I think some of answers may work for the DMDA.
> Now I have overcome this
> by creating a PetscSF and PetscSFBcast, and the procedures are described
> as follow.
>
>
> 1. Using DMGLobaltoLocal to update the local vector(the ghost cell mighb
> get the wrong result)
> 2. Creating a PetscSF which matches the local ghost cell and the global
> donor cell across the processes
> 3. Using PetscSFBCast to update the local ghost cell
>
> Now, my code can work. Can you give me a more scalable procedure?
>

That is a scalable procedure. I do not do it that way because it requires
more steps than just meshing the structure directly.

  Thanks,

    Matt


> Thanks.
>
> On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi all Petscer,
>         I have ask helps for some questions. Thanks for the replies
> from developer. I have known more about the petsc. I have also sought
> helps in the mail lists, and I find that there are many subjects
> focused on such a problem. Some subjects are listed as follow
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> ,
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
>
> ......
>
> The problem can be summarized as
> 1. Getting the value of vector from other processor
> 2. How to set the value of ghost cell, the value of the ghost cell
> might be as same as a very cell in other processor.
>
> I think the above problem is very popular. I suffer such a problem when
> I want to treat the periodic boundary for the FVM method.
>
>
> Ah, I think I am now seeing the problem. In DMPlex, we do not implement
> periodicity of the mesh by putting in extra communication. We make a
> periodic mesh directly. For example, in 1D what you describe would mesh
> a line segment, and then communicate the values from one end to the other.
> Instead, in Plex we just mesh a circle.
>
>   Thanks,
>
>     Matt
>
>
> After the dm
> object is distributed, the donor cell of bound might be in other
> processor. Since the donor cell must be matched as a pair correctly,
> using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get the
> expected results.
>
> In fact, such a problem is not very difficult for a mpi program. But we
> are coding with the petsc, we always think that whether we can
> implement more easily.
>
> Whether our developer can create a demo for us to following. I think
> such a demo is very useful for user.
>
> Thanks
>
> leejeal
>
>
>
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/2d69153c/attachment-0001.html>

From leejearl at mail.nwpu.edu.cn  Thu Jan 17 20:39:35 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Fri, 18 Jan 2019 10:39:35 +0800
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
	<d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
	<CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>
Message-ID: <88d61d2245e80220e6833a1a802aebf7ff7a6c08.camel@mail.nwpu.edu.cn>

Hi, Matt,



On Thu, 2019-01-17 at 21:20 -0500, Matthew Knepley wrote:
> On Thu, Jan 17, 2019 at 9:04 PM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
> > Hi Matt,
> > 	Thanks for your reply.  We have got to a consensus. For such a
> > problem, I have sought the mail list for helps and found a great 
> > many misleading answers. I think some of answers may work for the
> > DMDA. Now I have overcome this 
> > by creating a PetscSF and PetscSFBcast, and the procedures are
> > described as follow.
> > 
> > 
> > 1. Using DMGLobaltoLocal to update the local vector(the ghost cell
> > mighb get the wrong result)
> > 2. Creating a PetscSF which matches the local ghost cell and the
> > global donor cell across the processes
> > 3. Using PetscSFBCast to update the local ghost cell
> > 
> > Now, my code can work. Can you give me a more scalable procedure?
> 
> That is a scalable procedure. I do not do it that way because it
> requires more steps than just meshing the structure directly.

You are right. You have said that the more efficient way is to meshing
the structure directly. It means that we have better to generate 
a mesh with periodic topology structure. Is it right?

>   Thanks,
>     Matt 
> > Thanks.
> > 
> > On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
> > > On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> > > petsc-users at mcs.anl.gov> wrote:
> > > > Hi all Petscer,
> > > > 
> > > >         I have ask helps for some questions. Thanks for the
> > > > replies
> > > > 
> > > > from developer. I have known more about the petsc. I have also
> > > > sought
> > > > 
> > > > helps in the mail lists, and I find that there are many
> > > > subjects
> > > > 
> > > > focused on such a problem. Some subjects are listed as follow
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> > > > 
> > > > ,
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> > > > 
> > > > ,
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> > > > 
> > > > ,
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
> > > > 
> > > > 
> > > > 
> > > > ......
> > > > 
> > > > 
> > > > 
> > > > The problem can be summarized as
> > > > 
> > > > 1. Getting the value of vector from other processor
> > > > 
> > > > 2. How to set the value of ghost cell, the value of the ghost
> > > > cell
> > > > 
> > > > might be as same as a very cell in other processor. 
> > > > 
> > > > 
> > > > 
> > > > I think the above problem is very popular. I suffer such a
> > > > problem when
> > > > 
> > > > I want to treat the periodic boundary for the FVM method.
> > > 
> > > Ah, I think I am now seeing the problem. In DMPlex, we do not
> > > implementperiodicity of the mesh by putting in extra
> > > communication. We make aperiodic mesh directly. For example, in
> > > 1D what you describe would mesha line segment, and then
> > > communicate the values from one end to the other.Instead, in Plex
> > > we just mesh a circle.
> > >   Thanks,
> > >     Matt 
> > > >  After the dm
> > > > 
> > > > object is distributed, the donor cell of bound might be in
> > > > other
> > > > 
> > > > processor. Since the donor cell must be matched as a pair
> > > > correctly,
> > > > 
> > > > using the routines DMGlobaltoLocal and DMLocaltoGlobal can not
> > > > get the
> > > > 
> > > > expected results. 
> > > > 
> > > > 
> > > > 
> > > > In fact, such a problem is not very difficult for a mpi
> > > > program. But we
> > > > 
> > > > are coding with the petsc, we always think that whether we can
> > > > 
> > > > implement more easily.
> > > > 
> > > > 
> > > > 
> > > > Whether our developer can create a demo for us to following. I
> > > > think
> > > > 
> > > > such a demo is very useful for user.
> > > > 
> > > > 
> > > > 
> > > > Thanks
> > > > 
> > > > 
> > > > 
> > > > leejeal
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> 
> 
Thanks
leejearl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/fd706e6e/attachment.html>

From knepley at gmail.com  Thu Jan 17 20:41:19 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 21:41:19 -0500
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <88d61d2245e80220e6833a1a802aebf7ff7a6c08.camel@mail.nwpu.edu.cn>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
	<d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
	<CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>
	<88d61d2245e80220e6833a1a802aebf7ff7a6c08.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4G=M9T2Q5TqTfS_4UVRpZwkMXHqYHXeKH64FUhU51WVXZg@mail.gmail.com>

On Thu, Jan 17, 2019 at 9:39 PM leejearl <leejearl at mail.nwpu.edu.cn> wrote:

> Hi, Matt,
> On Thu, 2019-01-17 at 21:20 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 9:04 PM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
>
> Hi Matt,
> Thanks for your reply. We have got to a consensus. For such a problem, I
> have sought the mail list for helps and found a great
> many misleading answers. I think some of answers may work for the DMDA.
> Now I have overcome this
> by creating a PetscSF and PetscSFBcast, and the procedures are described
> as follow.
>
>
> 1. Using DMGLobaltoLocal to update the local vector(the ghost cell mighb
> get the wrong result)
> 2. Creating a PetscSF which matches the local ghost cell and the global
> donor cell across the processes
> 3. Using PetscSFBCast to update the local ghost cell
>
> Now, my code can work. Can you give me a more scalable procedure?
>
>
> That is a scalable procedure. I do not do it that way because it requires
> more steps than just meshing the structure directly.
>
>
> You are right. You have said that the more efficient way is to meshing the
> structure directly. It means that we have better to generate
> a mesh with periodic topology structure. Is it right?
>

Just to be clear, your method is perfectly fine. Mine is not more
efficient. I just think its easier.
You can see periodic meshes generated using DMPlexCreateBoxMesh().

  Thanks,

    Matt


>   Thanks,
>
>     Matt
>
>
> Thanks.
>
> On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi all Petscer,
>         I have ask helps for some questions. Thanks for the replies
> from developer. I have known more about the petsc. I have also sought
> helps in the mail lists, and I find that there are many subjects
> focused on such a problem. Some subjects are listed as follow
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> ,
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
>
> ......
>
> The problem can be summarized as
> 1. Getting the value of vector from other processor
> 2. How to set the value of ghost cell, the value of the ghost cell
> might be as same as a very cell in other processor.
>
> I think the above problem is very popular. I suffer such a problem when
> I want to treat the periodic boundary for the FVM method.
>
>
> Ah, I think I am now seeing the problem. In DMPlex, we do not implement
> periodicity of the mesh by putting in extra communication. We make a
> periodic mesh directly. For example, in 1D what you describe would mesh
> a line segment, and then communicate the values from one end to the other.
> Instead, in Plex we just mesh a circle.
>
>   Thanks,
>
>     Matt
>
>
> After the dm
> object is distributed, the donor cell of bound might be in other
> processor. Since the donor cell must be matched as a pair correctly,
> using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get the
> expected results.
>
> In fact, such a problem is not very difficult for a mpi program. But we
> are coding with the petsc, we always think that whether we can
> implement more easily.
>
> Whether our developer can create a demo for us to following. I think
> such a demo is very useful for user.
>
> Thanks
>
> leejeal
>
>
>
>
>
>
>
>
>
>
> Thanks
> leejearl
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/7c90f71c/attachment.html>

From leejearl at mail.nwpu.edu.cn  Thu Jan 17 20:48:11 2019
From: leejearl at mail.nwpu.edu.cn (leejearl)
Date: Fri, 18 Jan 2019 10:48:11 +0800
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <CAMYG4G=M9T2Q5TqTfS_4UVRpZwkMXHqYHXeKH64FUhU51WVXZg@mail.gmail.com>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
	<d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
	<CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>
	<88d61d2245e80220e6833a1a802aebf7ff7a6c08.camel@mail.nwpu.edu.cn>
	<CAMYG4G=M9T2Q5TqTfS_4UVRpZwkMXHqYHXeKH64FUhU51WVXZg@mail.gmail.com>
Message-ID: <9da0ce3c102f0ed139d3778618a90a5ea3b50b52.camel@mail.nwpu.edu.cn>

Hi Matt,	Thanks again for your helps. You have help me a lot.
Petsc is a great library. 	I need to learn more about it, thanks.
LeejearlOn Thu, 2019-01-17 at 21:41 -0500, Matthew Knepley wrote:
> On Thu, Jan 17, 2019 at 9:39 PM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
> > Hi, Matt,
> > On Thu, 2019-01-17 at 21:20 -0500, Matthew Knepley wrote:
> > > On Thu, Jan 17, 2019 at 9:04 PM leejearl <
> > > leejearl at mail.nwpu.edu.cn> wrote:
> > > > Hi Matt,
> > > > 	Thanks for your reply.  We have got to a consensus. For
> > > > such a problem, I have sought the mail list for helps and found
> > > > a great 
> > > > many misleading answers. I think some of answers may work for
> > > > the DMDA. Now I have overcome this 
> > > > by creating a PetscSF and PetscSFBcast, and the procedures are
> > > > described as follow.
> > > > 
> > > > 
> > > > 1. Using DMGLobaltoLocal to update the local vector(the ghost
> > > > cell mighb get the wrong result)
> > > > 2. Creating a PetscSF which matches the local ghost cell and
> > > > the global donor cell across the processes
> > > > 3. Using PetscSFBCast to update the local ghost cell
> > > > 
> > > > Now, my code can work. Can you give me a more scalable
> > > > procedure?
> > > 
> > > That is a scalable procedure. I do not do it that way because it
> > > requires more steps than just meshing the structure directly.
> > 
> > You are right. You have said that the more efficient way is to
> > meshing the structure directly. It means that we have better to
> > generate 
> > a mesh with periodic topology structure. Is it right?
> 
> Just to be clear, your method is perfectly fine. Mine is not more
> efficient. I just think its easier.You can see periodic meshes
> generated using DMPlexCreateBoxMesh().
>   Thanks,
>     Matt 
> > >   Thanks,
> > >     Matt 
> > > > Thanks.
> > > > 
> > > > On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
> > > > > On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> > > > > petsc-users at mcs.anl.gov> wrote:
> > > > > > Hi all Petscer,
> > > > > > 
> > > > > >         I have ask helps for some questions. Thanks for the
> > > > > > replies
> > > > > > 
> > > > > > from developer. I have known more about the petsc. I have
> > > > > > also sought
> > > > > > 
> > > > > > helps in the mail lists, and I find that there are many
> > > > > > subjects
> > > > > > 
> > > > > > focused on such a problem. Some subjects are listed as
> > > > > > follow
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> > > > > > 
> > > > > > ,
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> > > > > > 
> > > > > > ,
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> > > > > > 
> > > > > > ,
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > ......
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > The problem can be summarized as
> > > > > > 
> > > > > > 1. Getting the value of vector from other processor
> > > > > > 
> > > > > > 2. How to set the value of ghost cell, the value of the
> > > > > > ghost cell
> > > > > > 
> > > > > > might be as same as a very cell in other processor. 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > I think the above problem is very popular. I suffer such a
> > > > > > problem when
> > > > > > 
> > > > > > I want to treat the periodic boundary for the FVM method.
> > > > > 
> > > > > Ah, I think I am now seeing the problem. In DMPlex, we do not
> > > > > implementperiodicity of the mesh by putting in extra
> > > > > communication. We make aperiodic mesh directly. For example,
> > > > > in 1D what you describe would mesha line segment, and then
> > > > > communicate the values from one end to the other.Instead, in
> > > > > Plex we just mesh a circle.
> > > > >   Thanks,
> > > > >     Matt 
> > > > > >  After the dm
> > > > > > 
> > > > > > object is distributed, the donor cell of bound might be in
> > > > > > other
> > > > > > 
> > > > > > processor. Since the donor cell must be matched as a pair
> > > > > > correctly,
> > > > > > 
> > > > > > using the routines DMGlobaltoLocal and DMLocaltoGlobal can
> > > > > > not get the
> > > > > > 
> > > > > > expected results. 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > In fact, such a problem is not very difficult for a mpi
> > > > > > program. But we
> > > > > > 
> > > > > > are coding with the petsc, we always think that whether we
> > > > > > can
> > > > > > 
> > > > > > implement more easily.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Whether our developer can create a demo for us to
> > > > > > following. I think
> > > > > > 
> > > > > > such a demo is very useful for user.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Thanks
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > leejeal
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > 
> > > 
> > Thanks
> > leejearl
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/0f5fd94e/attachment-0001.html>

From knepley at gmail.com  Thu Jan 17 21:00:32 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Jan 2019 22:00:32 -0500
Subject: [petsc-users] I suggest to create a demo code for the problem
 of "Get a value from global vector to a local processor"
In-Reply-To: <9da0ce3c102f0ed139d3778618a90a5ea3b50b52.camel@mail.nwpu.edu.cn>
References: <b901c246d54de466d472c0e26c11f3bb8a76615b.camel@mail.nwpu.edu.cn>
	<CAMYG4G=QDdJcpOpzymnmAuOPfMkztmcjoXRgCsGPNbyWj6F_bw@mail.gmail.com>
	<d39f0214aa125feabcf81656cb16fb75bbacd95d.camel@mail.nwpu.edu.cn>
	<CAMYG4GnvwknuAHadtdV=XqQ1EW2zfo1+1G4ydCFnw+XnbCC+wA@mail.gmail.com>
	<88d61d2245e80220e6833a1a802aebf7ff7a6c08.camel@mail.nwpu.edu.cn>
	<CAMYG4G=M9T2Q5TqTfS_4UVRpZwkMXHqYHXeKH64FUhU51WVXZg@mail.gmail.com>
	<9da0ce3c102f0ed139d3778618a90a5ea3b50b52.camel@mail.nwpu.edu.cn>
Message-ID: <CAMYG4G=g9xXdZMqhOfJUMNuuLAdjxu9Tpx9M4UNqiAFqb0txXw@mail.gmail.com>

On Thu, Jan 17, 2019 at 9:48 PM leejearl <leejearl at mail.nwpu.edu.cn> wrote:

> Hi Matt,
> Thanks again for your helps. You have help me a lot. Petsc is a great
> library.
> I need to learn more about it, thanks.
>

Cool. I am glad you got it working. Don't hesitate to suggest functionality
you think we need as well.

  Thanks,

    Matt


> Leejearl
> On Thu, 2019-01-17 at 21:41 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 9:39 PM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
>
> Hi, Matt,
> On Thu, 2019-01-17 at 21:20 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 9:04 PM leejearl <leejearl at mail.nwpu.edu.cn>
> wrote:
>
> Hi Matt,
> Thanks for your reply. We have got to a consensus. For such a problem, I
> have sought the mail list for helps and found a great
> many misleading answers. I think some of answers may work for the DMDA.
> Now I have overcome this
> by creating a PetscSF and PetscSFBcast, and the procedures are described
> as follow.
>
>
> 1. Using DMGLobaltoLocal to update the local vector(the ghost cell mighb
> get the wrong result)
> 2. Creating a PetscSF which matches the local ghost cell and the global
> donor cell across the processes
> 3. Using PetscSFBCast to update the local ghost cell
>
> Now, my code can work. Can you give me a more scalable procedure?
>
>
> That is a scalable procedure. I do not do it that way because it requires
> more steps than just meshing the structure directly.
>
>
> You are right. You have said that the more efficient way is to meshing the
> structure directly. It means that we have better to generate
> a mesh with periodic topology structure. Is it right?
>
>
> Just to be clear, your method is perfectly fine. Mine is not more
> efficient. I just think its easier.
> You can see periodic meshes generated using DMPlexCreateBoxMesh().
>
>   Thanks,
>
>     Matt
>
>
>   Thanks,
>
>     Matt
>
>
> Thanks.
>
> On Thu, 2019-01-17 at 06:43 -0500, Matthew Knepley wrote:
>
> On Thu, Jan 17, 2019 at 3:34 AM leejearl via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi all Petscer,
>         I have ask helps for some questions. Thanks for the replies
> from developer. I have known more about the petsc. I have also sought
> helps in the mail lists, and I find that there are many subjects
> focused on such a problem. Some subjects are listed as follow
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037425.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-January/024068.html
> ,
>
>
>
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-November/003633.html
> ,
>
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2019-January/037436.html
>
> ......
>
> The problem can be summarized as
> 1. Getting the value of vector from other processor
> 2. How to set the value of ghost cell, the value of the ghost cell
> might be as same as a very cell in other processor.
>
> I think the above problem is very popular. I suffer such a problem when
> I want to treat the periodic boundary for the FVM method.
>
>
> Ah, I think I am now seeing the problem. In DMPlex, we do not implement
> periodicity of the mesh by putting in extra communication. We make a
> periodic mesh directly. For example, in 1D what you describe would mesh
> a line segment, and then communicate the values from one end to the other.
> Instead, in Plex we just mesh a circle.
>
>   Thanks,
>
>     Matt
>
>
> After the dm
> object is distributed, the donor cell of bound might be in other
> processor. Since the donor cell must be matched as a pair correctly,
> using the routines DMGlobaltoLocal and DMLocaltoGlobal can not get the
> expected results.
>
> In fact, such a problem is not very difficult for a mpi program. But we
> are coding with the petsc, we always think that whether we can
> implement more easily.
>
> Whether our developer can create a demo for us to following. I think
> such a demo is very useful for user.
>
> Thanks
>
> leejeal
>
>
>
>
>
>
>
>
>
>
> Thanks
> leejearl
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190117/76713015/attachment.html>

From jczhang at mcs.anl.gov  Thu Jan 17 21:12:39 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Fri, 18 Jan 2019 03:12:39 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
Message-ID: <CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>

Usually when I meet a SEGV error, I will run it again with a parallel debugger like DDT and wait for it to segfault, and then examine the stack trace to see what is wrong.
Can you compress your matrix and upload it to google drive, so we can try to reproduce the error.
--Junchao Zhang


On Thu, Jan 17, 2019 at 10:44 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I did two runs, one with the SuperLU_dist and one with bcgs using jacobi, attached are the results of one of the reports from valgrind on one random processor (out of the 128 files).

DS = direct solver
IS = iterative  solver

There is an awful lot of errors.

how I initiated the two runs:
mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce -ksp_monitor -log_view

mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

Thank you

On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
1) Running out of memory

2) You passed an invalid array
I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e. using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors per node (128 processors in total).
I am not sure what would constitute an invalid array or how I can check that. I am using the same procedure as when dealing with the smaller matrix. i.e. Generate matrix A and vector b using FEM software then convert the matrix and vector using a python script ready for petsc. read in petsc and calculate.

Are you running with 64-bit ints here?
Yes I have it configured petsc with  --with-64-bit-indices and debugging mode, which this was run on.

It sounds like you have enough memory, but the fact that is runs for smaller problems makes me suspicious. It
could still be a memory overwrite. Can you either

a) Run under valgrind

or

b) Run under the debugger and get a stack trace

 ?

   Thanks,

    Matt

On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
SuperLU_dist supports 64-bit ints. Are you not running in parallel?
I will try that, although I think solving the real problem (later on if I can get this to work) with 30 million finite elements might be a problem for SuperLU_dist. so it is better to get an iterative solver to work with first.

1) Try using    -build_twosided allreduce    on this run
How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce -ksp_monitor -log_view
I have attached the full error output.

You are getting an SEGV on MatSetValues(), so its either

1) Running out of memory

2) You passed an invalid array

2) Is it possible to get something that fails here but we can run. None of our tests show this problem.
I am not how I can do that, but i have added my code which is quite short and should only read and solve the system, the problem arises at larger matrices for example current test case has 6 million finite elements (~2B non-zero numbers and 25M x 25M matrix).

Are you running with 64-bit ints here?

   Matt

On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
The memory requested is an insane number. You may need to use 64 bit integers.
Thanks Mark, I reconfigured it to use 64bit, however in the process it says I can no longer use MUMPS and SuperLU as they are not supported (I see on MUMPS webpage it supports 64int). However it does not exactly solve the problem.

SuperLU_dist supports 64-bit ints. Are you not running in parallel?

This time, it crashes at
[6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
ierr      = PetscMalloc1(bi[pn]+1,&bj);
which allocates local portion of B^T*A.
You may also try to increase number of cores to reduce local matrix size.

So I increased the number of cores to 16 on one node and ran it by :
mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
It crashed after reading in the matrix and before starting to solve. The error:

[15]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[3]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR: ------------------------------------------------------------------------
[8]PETSC ERROR: ------------------------------------------------------------------------
[12]PETSC ERROR: ------------------------------------------------------------------------
[12]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------
[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
--------------------------------------------------------------------------
mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on signal 9 (Killed).

Now I was running this with valgrind as someone had previously suggested and the 16 files created all contain the same type of error:

Okay, its possible that there are bugs in the MPI implementation. So

1) Try using    -build_twosided allreduce    on this run

2) Is it possible to get something that fails here but we can run. None of our tests show this problem.

  Thanks,

     Matt

==25940== Invalid read of size 8
==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:373)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16 alloc'd
==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:371)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==


On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Fande:
According to this PR https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff

Should we set the scalable algorithm as default?
Sure, we can. But I feel we need do more tests to compare scalable and non-scalable algorithms.
On theory, for small to medium matrices, non-scalable matmatmult() algorithm enables more efficient
data accessing. Andreas optimized scalable implementation. Our non-scalable implementation might have room to be further optimized.
Hong

On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Add option '-mattransposematmult_via scalable'
Hong

On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/0f262eca/attachment-0001.html>

From rickcha at googlemail.com  Fri Jan 18 06:52:54 2019
From: rickcha at googlemail.com (rickcha at googlemail.com)
Date: Fri, 18 Jan 2019 13:52:54 +0100
Subject: [petsc-users] Error in DMFieldComputeFaceData_DS with hex elements
Message-ID: <2415BCFF-0A48-4181-8B80-8B85D62C6F21@googlemail.com>

Dear petsc-team,

I ran into an error when using petscFE in combination with a hex mesh constructed with gmsh. The error message reads:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Not implemented yet

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5  GIT Date: 2019-01-17 05:31:34 +0100
[0]PETSC ERROR: ./standaloneFEM on a arch-linux2-intel-debug named iket127138 by hartig Fri Jan 18 10:49:52 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel-debug --with-debugging=true --with-cc=/home/hartig/intel/bin/icc --with-cxx=/home/hartig/intel/bin/icpc --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml --download-parmetis --download-metis --download-hypre --download-mumps --download-scalapack --download-triangle --download-hdf5 --download-ptscotch --download-chaco
[0]PETSC ERROR: #1 DMFieldComputeFaceData_DS() line 943 in /home/hartig/petsc/src/dm/field/impls/ds/dmfieldds.c
[0]PETSC ERROR: #2 DMFieldCreateFEGeom() line 537 in /home/hartig/petsc/src/dm/field/interface/dmfield.c
[0]PETSC ERROR: #3 DMSNESGetFEGeom() line 2598 in /home/hartig/petsc/src/dm/impls/plex/plexfem.c
[0]PETSC ERROR: #4 DMPlexComputeBdResidual_Single_Internal() line 1520 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
[0]PETSC ERROR: #5 DMPlexComputeBdResidual_Internal() line 1649 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
[0]PETSC ERROR: #6 DMPlexComputeResidual_Internal() line 1946 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
[0]PETSC ERROR: #7 DMPlexTSComputeIFunctionFEM() line 206 in /home/hartig/petsc/src/ts/utils/dmplexts.c
[0]PETSC ERROR: #8 ComputeI2FunctionWithInertia() line 1284 in /home/hartig/cimply/femanalysis.c
[0]PETSC ERROR: #9 TSComputeI2Function() line 1707 in /home/hartig/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #10 SNESTSFormFunction_Alpha() line 339 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
[0]PETSC ERROR: #11 SNESTSFormFunction() line 4694 in /home/hartig/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #12 SNESComputeFunction() line 2301 in /home/hartig/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 175 in /home/hartig/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #14 SNESSolve() line 4454 in /home/hartig/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #15 TSAlpha_SNESSolve() line 101 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
[0]PETSC ERROR: #16 TSAlpha_Restart() line 143 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
[0]PETSC ERROR: #17 TSStep_Alpha() line 217 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
[0]PETSC ERROR: #18 TSStep() line 3664 in /home/hartig/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #19 TSSolve() line 3847 in /home/hartig/petsc/src/ts/interface/ts.c



Upon investigation, I found that petsc throws the error message because coneSize has a value of six. This is the first thing that puzzles me. I would have expected a cone size of 4 since we are dealing with faces of hex elements. Is it possible that we are on the wrong ?level? of the mesh graph somehow? Printing "numFaces? in the debugger, I get the total number of elements (not 2D faces) I have in my mesh for the hex-case.

I ran the same case with a simplex mesh and found another thing I don?t quite understand: Both times I ran with -petscspace-degree 2. Yet in case of the simplex mesh, I get a polynomial degree of 1 in DMFieldComputeFaceData_DS (?maxDegree?) and in case of the hex-mesh, I get a polynomial degree of 3. In both cases I would have expected the degree to be 2, corresponding to Tet10 and Hex20 elements. I might be confusing two different concepts here. But this was how I understood the documentation until now.

Thanks,
Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/ced1b8ad/attachment.html>

From nils.schween at uni-heidelberg.de  Fri Jan 18 07:23:46 2019
From: nils.schween at uni-heidelberg.de (Nils Schween)
Date: Fri, 18 Jan 2019 14:23:46 +0100
Subject: [petsc-users] ASM Interface (Additive Schwarz Method)
Message-ID: <871s5a5cbh.fsf@uni-heidelberg.de>

Hello everyone,

after working a while with Petsc?s ASM preconditioner, I ended up a little bit
confused.

I drew a picture to demonstrate to you how I am setting up the ASM
blocks via the routine PCASMSetLocalSubdomains().

	  Indexset is_local                                                                             idxArrayASMBlockLocal: [1,...,60,...,90,...]
        1 +--------------------------------------------------+...                 |                    ^      ^     ^
          |                                                  |  .. rank 0 --------+--------------------|      |     |
          |                                                  |...                 |                           |     |
          |                                                  |...                 |                           |     |
          |                                                  |  .. rank 1         |                           |     |
          |                                                  |...                 |                           |     |
          |                                                  |                    |                           |     |
          |                                                  |                    |                           |     |
          |                                                  | .                  |                           |     |
          |                                                  | .                  |                           |     |
          |                                                  | .                  |                           |     |
          |                                                  |                    |                           |     |
          |                                                  |                    |                           |     |
          |                                                  |...                 |                           |     |
          |                                                  |  .. rank size-1    |                           |     |
          |                                                  |...                 |                           |     |
       60 +--------------------------------------------------+...                 |                           |     |
          |                                                  |  .. rank 0 --------+---------------------------+     |
          |                                                  |...                 |                                 |
          |                                                  | .                  |                                 |
          |                                                  | .                  |                                 |
          |                                                  | .                  |                                 |
          |                                                  |                    |                                 |
       90 +--------------------------------------------------+ ...                |                                 |
          |                                                  |   .. rank 0--------+---------------------------------+
          |                                                  | ...                v
          |                                                  |  .   ISCreateGeneral(PETSC_COMM_WORLD, numberOfElements,idxArrayAsMBlockLocal, PETSC_USE_POINTER, &IndicesASMBlockLocal);
      100 +--------------------------------------------------+  .

      	  Indexset is         					      		      idxArrayASMBlock: [1,...,olap,60,...,olap,90,...]
	    									     |		    ^		^	    ^
        1 +--------------------------------------------------+...		     |		    |		|	    |
          |                                                  |  .		     | 		    |		|	    |
          |                                                  +--+- rank 0 -----------+--------------|		|	    |
          | 						     | 	|		     | 				|	    |
          |                                                  |..|		     | 				|	    |
          |                                                  |	+- rank 1	     | 			        |	    |
          |                                                  |	|		     | 			        |	    |
          |                                                  |	|		     | 			        |	    |
          |    	       	       	       	       	       	     +--+		     | 			        |	    |
          |                                                  | .		     | 			        |	    |
          |                                                  | .		     | 			        |	    |
          |                                                  |...		     | 			        |	    |
          |                                                  | 	.		     | 			        |	    |
          |                                                  | 	.. rank size-1	     | 			        |	    |
          |                                                  |  .		     |  		        |	    |
          |                                                  |...		     |  		        |	    |
       60 +--------------------------------------------------+...		     |  		        |	    |
          |                                                  |. .		     |  			|	    |
          |                                                  +.-+. rank 0 -----------+--------------------------+	    |
          |                                                  |. |		     |  				    |
          |                                                  |..|		     |  				    |
          |                                                  |  |		     | 					    |
          |                                                  |  +- rank 1	     | 					    |
       90 +--------------------------------------------------+...		     | 					    |
          |                                                  |  .		     | 					    |
          |   	       	       	       	       	       	     +--+. rank 0 -----------+--------------------------------------+
          |                                                  |  |		     |
      100 +--------------------------------------------------+ .|		     v
       	       	       	       	       	       	       	       	+- rank 1  ISCreateGeneral(PETSC_COMM_WORD, numberOfElements+elementsOlap, idxArrayASMBlock, PETSC_USE_POINTER, &IndicesASMBlockLocal);

The PCASMSetLocalSubdomains() routine requires, next to other arguments, two
index sets. The above drawing shows how these index sets are obtained: Think
of the matrix to be preconditioned as split into three parts. To create an ASM
block on one processor, say rank 0, rows from each part are needed. (As can be
seen in the first part of the above picture). The row indices of the needed rows
are stored in an integer array (i.e. in my case idxArrayASMBlockLocal or
idxArrayASMBlock) which is used to create an index set with the help of
ISCreateGeneral(). The second part of the above picture demonstrates what row
indices are used to create the overlap.

This works perfectly as long as I use PC_ASM_BASIC for the PCASMType. If I
switch to PC_ASM_RESTRICT, my GMRES algorithm, does not converge anymore.
Why is this?
Am I misunderstanding the ASM interface? (i.e. is the way I define the overlap
wrong?)
Do I have to somehow permute the right-hand side of system of equations?

It?s probably important to know, that PCSide is set to PC_RIGHT.

Thank you for your help!
With best regards,
Nils

P.S.: I very much hope, that my small picture is not destroyed by our email clients.


--
M.Sc. Nils Schween
Research Associate

Phone: +49 6221 54 14512
Mail: nils.schween at uni-heidelberg.de

Heidelberg University
Engineering Mathematics and Computing Lab (EMCL)
Interdisciplinary Center for Scientific Computing (IWR)
Im Neuenheimer Feld 205, D-69120 Heidelberg
http://emcl.iwr.uni-heidelberg.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/8b39ea0b/attachment.sig>

From mfadams at lbl.gov  Fri Jan 18 07:42:19 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 18 Jan 2019 08:42:19 -0500
Subject: [petsc-users] ASM Interface (Additive Schwarz Method)
In-Reply-To: <871s5a5cbh.fsf@uni-heidelberg.de>
References: <871s5a5cbh.fsf@uni-heidelberg.de>
Message-ID: <CADOhEh7DAnxeTw2fu=FKoLPMRRkwEswmSpVkZ49tg32B+gKHyA@mail.gmail.com>

>
>
>
> This works perfectly as long as I use PC_ASM_BASIC for the PCASMType. If I
> switch to PC_ASM_RESTRICT, my GMRES algorithm, does not converge anymore.
> Why is this?
>

PC_ASM_RESTRICT specifies the use of a different algorithm. It is a cheaper
algorithm in terms of work (communication) but is weaker mathematically, in
parallel.


> Am I misunderstanding the ASM interface? (i.e. is the way I define the
> overlap
> wrong?)
> Do I have to somehow permute the right-hand side of system of equations?
>
> It?s probably important to know, that PCSide is set to PC_RIGHT.
>
> Thank you for your help!
> With best regards,
> Nils
>
> P.S.: I very much hope, that my small picture is not destroyed by our
> email clients.
>
>
> --
> M.Sc. Nils Schween
> Research Associate
>
> Phone: +49 6221 54 14512
> Mail: nils.schween at uni-heidelberg.de
>
> Heidelberg University
> Engineering Mathematics and Computing Lab (EMCL)
> Interdisciplinary Center for Scientific Computing (IWR)
> Im Neuenheimer Feld 205, D-69120 Heidelberg
> http://emcl.iwr.uni-heidelberg.de
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/a3151823/attachment.html>

From valerio.barnabei at gmail.com  Fri Jan 18 08:53:53 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Fri, 18 Jan 2019 15:53:53 +0100
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
Message-ID: <CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>

Hello,
I'm trying to figure out how to change my program in order to use MATIS
type and PCBDD.
I managed to build my MATIS matrix successfully, but my code is now slower.
I think the main issue is that i'm using MatSetValues inside the element
loop, and passing as values the e-th local element matrix. In a sort of
pseudocode:

for e in localElements:{
for (int qp=0;qp<ngp_tot;qp++)
       for(int A=0;A<Nnodesloc;A++){
                for(int B=0;B<Nnodesloc;B++){
                    //Bat dot temp (NB: ba_n0=bat_0n)
                    double
prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
                    //ke[A][B]+=weights[qp]*detJ[qp]*prod;
                    ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;


?
Actually i have lots of doubts:
-when using ISLocalToGlobalMappingCreate(), what's the blocksize?

Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <
valerio.barnabei at gmail.com> ha scritto:

> Thank you Stefano for your quick answer, I will try to change PC as you
> suggest, and I will read something about MATIS type. I will let you know if
> I still have doubts.
>
> Valerio
>
> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <
> stefano.zampini at gmail.com> ha scritto:
>
>> What problem are you trying to solve?
>> For standard elliptic problems (e.g. poisson or elasticity) you should
>> use a preconditioner that scales, for example PCGAMG or BoomerAMG from
>> HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via
>> --download-hypre)
>> Alternatively, since you are using FEM with local Assembly, you can use
>> PCBDDC, but this requires few extra calls
>>
>> - call MatSetType(K,MATIS)
>> - provide an ISLocalToGlobalMapping of local to global degrees of freedom
>> before you set preallocation via MatSetLocalToGlobalMapping (if you provide
>> this call, then you can use MatSetValuesLocal for both AIJ amd MATIS types)
>>
>>
>>
>> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via petsc-users
>> <petsc-users at mcs.anl.gov> ha scritto:
>>
>>> Hello everybody,
>>>
>>> I have some doubts about the parallel solution of a simple FEM ksp
>>> problem. I searched in the mailing list but the information I found are not
>>> helping me.
>>> I need to control manually the parallel assemble of the main matrix, but
>>> I would like to use PetSC for the solution of the final linear system
>>> K*U=F.
>>> The algorithm I am using, loads an already decomposed FEM domain, and
>>> each process contributes to the global matrix assembly using its portion of
>>> domain decomposition (made in pre-processing). In the element loop, at each
>>> element I load the element submatrix in the global matrix via MatSetValue
>>> function. Then, out of the element loop, I write
>>> /*
>>> ...matrix preallocation
>>> ...calculate element local stiffness matrix (local in a FEM sense and
>>> local in a domain decomposition sense)
>>> MatSetValue (ADD)
>>> */
>>> and then
>>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>>>
>>> This part scales perfectly and is sufficiently fast.
>>> My K matrix is a sort of banded diagonal matrix: my preallocation is not
>>> perfect, but it works at this stage of my work.
>>> Finally I solve the system with the ksp logic:
>>>
>>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI,
>>> the behaviour is almost identical, the iteration number lowers but is still
>>> really high
>>>
>>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>> ierr=KSPSolve(ksp,F,sol);
>>>
>>> When I run the code I am noticing some strange behaviours. Indeed, the
>>> numerical solution appear right, and the code scales if I make a simple
>>> strong-scaling test. However, the ksp_log and log_view (see below, logs for
>>> a 160k quad linear element uniform 2dmesh) shows me a huge number of
>>> iterations, and the time spent in "solve" function is very high.
>>> Furthermore, the more I increase the problem size keeping a constant load
>>> per process (weak scaling) the more iterations seems to be needed for
>>> convergence. In general, when increasing the problem size (i.e. 16M
>>> elements) the convergence is not achieved before the maximum iteration
>>> number is achieved.
>>>
>>> Question 1: Am I doing it right? Or am I making some trouble with the
>>> MPI communication management? (i'm aware that my domain decomposition does
>>> not match the petsc matrix decomposition, I'm expecting it to cripple my
>>> algorithm a bit)
>>> Question 2: Why the solver takes so much time and so many iterations?
>>> the problem is really simple, and the solution appears correct when
>>> plotted. Am I ill conditioning the system?
>>> Thanks in advance for your help. I can add any further information if
>>> needed.
>>>
>>> Valerio
>>>
>>
>>
>> --
>> Stefano
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/4a41dc4e/attachment-0001.html>

From valerio.barnabei at gmail.com  Fri Jan 18 08:55:22 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Fri, 18 Jan 2019 15:55:22 +0100
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
	<CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
Message-ID: <CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>

The mail was still incomplete. Sorry, i will write it again.
Apologies.

Il giorno ven 18 gen 2019 alle ore 15:53 Valerio Barnabei <
valerio.barnabei at gmail.com> ha scritto:

> Hello,
> I'm trying to figure out how to change my program in order to use MATIS
> type and PCBDD.
> I managed to build my MATIS matrix successfully, but my code is now
> slower. I think the main issue is that i'm using MatSetValues inside the
> element loop, and passing as values the e-th local element matrix. In a
> sort of pseudocode:
>
> for e in localElements:{
> for (int qp=0;qp<ngp_tot;qp++)
>        for(int A=0;A<Nnodesloc;A++){
>                 for(int B=0;B<Nnodesloc;B++){
>                     //Bat dot temp (NB: ba_n0=bat_0n)
>                     double
> prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
>                     //ke[A][B]+=weights[qp]*detJ[qp]*prod;
>                     ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;
>
>
> ?
> Actually i have lots of doubts:
> -when using ISLocalToGlobalMappingCreate(), what's the blocksize?
>
> Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <
> valerio.barnabei at gmail.com> ha scritto:
>
>> Thank you Stefano for your quick answer, I will try to change PC as you
>> suggest, and I will read something about MATIS type. I will let you know if
>> I still have doubts.
>>
>> Valerio
>>
>> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <
>> stefano.zampini at gmail.com> ha scritto:
>>
>>> What problem are you trying to solve?
>>> For standard elliptic problems (e.g. poisson or elasticity) you should
>>> use a preconditioner that scales, for example PCGAMG or BoomerAMG from
>>> HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via
>>> --download-hypre)
>>> Alternatively, since you are using FEM with local Assembly, you can use
>>> PCBDDC, but this requires few extra calls
>>>
>>> - call MatSetType(K,MATIS)
>>> - provide an ISLocalToGlobalMapping of local to global degrees of
>>> freedom before you set preallocation via MatSetLocalToGlobalMapping (if you
>>> provide this call, then you can use MatSetValuesLocal for both AIJ
>>> amd MATIS types)
>>>
>>>
>>>
>>> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via
>>> petsc-users <petsc-users at mcs.anl.gov> ha scritto:
>>>
>>>> Hello everybody,
>>>>
>>>> I have some doubts about the parallel solution of a simple FEM ksp
>>>> problem. I searched in the mailing list but the information I found are not
>>>> helping me.
>>>> I need to control manually the parallel assemble of the main matrix,
>>>> but I would like to use PetSC for the solution of the final linear system
>>>> K*U=F.
>>>> The algorithm I am using, loads an already decomposed FEM domain, and
>>>> each process contributes to the global matrix assembly using its portion of
>>>> domain decomposition (made in pre-processing). In the element loop, at each
>>>> element I load the element submatrix in the global matrix via MatSetValue
>>>> function. Then, out of the element loop, I write
>>>> /*
>>>> ...matrix preallocation
>>>> ...calculate element local stiffness matrix (local in a FEM sense and
>>>> local in a domain decomposition sense)
>>>> MatSetValue (ADD)
>>>> */
>>>> and then
>>>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>>>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>>>>
>>>> This part scales perfectly and is sufficiently fast.
>>>> My K matrix is a sort of banded diagonal matrix: my preallocation is
>>>> not perfect, but it works at this stage of my work.
>>>> Finally I solve the system with the ksp logic:
>>>>
>>>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>>>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>>>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI,
>>>> the behaviour is almost identical, the iteration number lowers but is still
>>>> really high
>>>>
>>>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>>>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>>> ierr=KSPSolve(ksp,F,sol);
>>>>
>>>> When I run the code I am noticing some strange behaviours. Indeed, the
>>>> numerical solution appear right, and the code scales if I make a simple
>>>> strong-scaling test. However, the ksp_log and log_view (see below, logs for
>>>> a 160k quad linear element uniform 2dmesh) shows me a huge number of
>>>> iterations, and the time spent in "solve" function is very high.
>>>> Furthermore, the more I increase the problem size keeping a constant load
>>>> per process (weak scaling) the more iterations seems to be needed for
>>>> convergence. In general, when increasing the problem size (i.e. 16M
>>>> elements) the convergence is not achieved before the maximum iteration
>>>> number is achieved.
>>>>
>>>> Question 1: Am I doing it right? Or am I making some trouble with the
>>>> MPI communication management? (i'm aware that my domain decomposition does
>>>> not match the petsc matrix decomposition, I'm expecting it to cripple my
>>>> algorithm a bit)
>>>> Question 2: Why the solver takes so much time and so many iterations?
>>>> the problem is really simple, and the solution appears correct when
>>>> plotted. Am I ill conditioning the system?
>>>> Thanks in advance for your help. I can add any further information if
>>>> needed.
>>>>
>>>> Valerio
>>>>
>>>
>>>
>>> --
>>> Stefano
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/e50cb3ed/attachment.html>

From knepley at gmail.com  Fri Jan 18 09:09:02 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 Jan 2019 10:09:02 -0500
Subject: [petsc-users] Error in DMFieldComputeFaceData_DS with hex
 elements
In-Reply-To: <2415BCFF-0A48-4181-8B80-8B85D62C6F21@googlemail.com>
References: <2415BCFF-0A48-4181-8B80-8B85D62C6F21@googlemail.com>
Message-ID: <CAMYG4GkAjzNx-Rpe8QF2cdbx9dwog+vUeqQ6+Jv28H6zRnCDow@mail.gmail.com>

On Fri, Jan 18, 2019 at 7:54 AM rickcha--- via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear petsc-team,
>
> I ran into an error when using petscFE in combination with a hex mesh
> constructed with gmsh. The error message reads:
>

Yes, I am guessing that GMsh put some crazy crap in the mesh file. Can you
test it out on a hex mesh from DMPlexCreateBoxMesh()
just ot make sure your physics works? Then we can look at the GMsh file (I
hate that program).


> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Not implemented yet
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5
>  GIT Date: 2019-01-17 05:31:34 +0100
> [0]PETSC ERROR: ./standaloneFEM on a arch-linux2-intel-debug named
> iket127138 by hartig Fri Jan 18 10:49:52 2019
> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel-debug
> --with-debugging=true --with-cc=/home/hartig/intel/bin/icc
> --with-cxx=/home/hartig/intel/bin/icpc
> --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml
> --download-parmetis --download-metis --download-hypre --download-mumps
> --download-scalapack --download-triangle --download-hdf5
> --download-ptscotch --download-chaco
> [0]PETSC ERROR: #1 DMFieldComputeFaceData_DS() line 943 in
> /home/hartig/petsc/src/dm/field/impls/ds/dmfieldds.c
> [0]PETSC ERROR: #2 DMFieldCreateFEGeom() line 537 in
> /home/hartig/petsc/src/dm/field/interface/dmfield.c
> [0]PETSC ERROR: #3 DMSNESGetFEGeom() line 2598 in
> /home/hartig/petsc/src/dm/impls/plex/plexfem.c
> [0]PETSC ERROR: #4 DMPlexComputeBdResidual_Single_Internal() line 1520 in
> /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #5 DMPlexComputeBdResidual_Internal() line 1649 in
> /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #6 DMPlexComputeResidual_Internal() line 1946 in
> /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #7 DMPlexTSComputeIFunctionFEM() line 206 in
> /home/hartig/petsc/src/ts/utils/dmplexts.c
> [0]PETSC ERROR: #8 ComputeI2FunctionWithInertia() line 1284 in
> /home/hartig/cimply/femanalysis.c
> [0]PETSC ERROR: #9 TSComputeI2Function() line 1707 in
> /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #10 SNESTSFormFunction_Alpha() line 339 in
> /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #11 SNESTSFormFunction() line 4694 in
> /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #12 SNESComputeFunction() line 2301 in
> /home/hartig/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 175 in
> /home/hartig/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #14 SNESSolve() line 4454 in
> /home/hartig/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #15 TSAlpha_SNESSolve() line 101 in
> /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #16 TSAlpha_Restart() line 143 in
> /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #17 TSStep_Alpha() line 217 in
> /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #18 TSStep() line 3664 in
> /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #19 TSSolve() line 3847 in
> /home/hartig/petsc/src/ts/interface/ts.c
>
> Upon investigation, I found that petsc throws the error message because
> coneSize has a value of six. This is the first thing that puzzles me. I
> would have expected a cone size of 4 since we are dealing with faces of hex
> elements. Is it possible that we are on the wrong ?level? of the mesh graph
> somehow? Printing "numFaces? in the debugger, I get the total number of
> elements (not 2D faces) I have in my mesh for the hex-case.
>
> I ran the same case with a simplex mesh and found another thing I don?t
> quite understand: Both times I ran with -petscspace-degree 2. Yet in case
> of the simplex mesh, I get a polynomial degree of 1 in
> DMFieldComputeFaceData_DS (?maxDegree?)
>

I think you are looking at the coordinate space. By default, simplex mesh
use affine coordinate spaces.


> and in case of the hex-mesh, I get a polynomial degree of 3.
>

That is strange. It should be 2.

  Thanks,

    Matt


> In both cases I would have expected the degree to be 2, corresponding to
> Tet10 and Hex20 elements. I might be confusing two different concepts here.
> But this was how I understood the documentation until now.
>
> Thanks,
> Max
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/f0e18dde/attachment.html>

From valerio.barnabei at gmail.com  Fri Jan 18 09:47:50 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Fri, 18 Jan 2019 16:47:50 +0100
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
	<CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
	<CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>
Message-ID: <CANZLC_fAqa1DyCvU=JD8SkgyaYj-bnukpy8Pmxi+6CjWB1e4NQ@mail.gmail.com>

Hello (again),

I'm trying to figure out how to change my program in order to use MATIS
type and PCBDD.

I managed to build my MATIS matrix successfully, but my code is now slower.
I think the main issue is that i'm using MatSetValues inside the element
loop, and passing as values the e-th local element matrix.

In a sort of pseudocode, where B is the shape functions matrix and D is a
?diffusion? matrix:



for e in localElements

for qp in quadraturePoints{

                        for n in localNodes {
                                   for m in localNodes {
                                               ke +=
weights[qp]*detJ[qp]*B[n]^T* D*B[m];

}

}

For n in localNodes{

For m in localNodes{

                fe[n] -= ke[n* localNodes +m]*ge[e][m];//ge is a simple
array that lets me easily take into account Dirichlet conditions, for my
simple case

}

for n in localNodes{

//            idx [n]=LM [e][n];

//LM is the DOF array (rows are Local elements, columns are local nodes,
values are global dofs, i.e. row/col indices for global assembly of K,
negative values are prescribed DOFs

}

MatSetValues(K, localNodes, (PetscInt*) &idx [0], localNodes, (PetscInt*)
&idx [0], (PetscScalar*) &ke[0], ADD_VALUES);

//same for VecSetValues

}

Just to clarify: the array I used to create the map is a unique array for
each process, without the negative values of the prescribed DOFs; the array
I?m using in matsetvalues is each time the portion of the array used for
the map relative to the current element. In addition to that: the code is
working, it gives the same result of an older version where I used MATAIJ,
but the portion of the element assembly loop is slower, and so is the
solving process (compared using PCNONE on both, obtained same iterations to
converge, same norm_2 of solution vector).

I guess using MatSetValues outside of the loop, using the same large array
I used to create the map for the matIS is supposed to be better, but I
cannot see a way to use it with this elemental assembly logic.

Stefano suggested to use MatSetValuesLocal: in that case what is supposed
to be idx? (I?m using global indices now).

If I use MatSetValuesLocal, can I stick with this logic of adding values
inside the element loop? Is there any other way you suggest to assign
values for this case?Any suggestion would be really appreciated.

I?m not sure I explained my issue properly, I can give any further
information needed.


Valerio



Il giorno ven 18 gen 2019 alle ore 15:55 Valerio Barnabei <
valerio.barnabei at gmail.com> ha scritto:

> The mail was still incomplete. Sorry, i will write it again.
> Apologies.
>
> Il giorno ven 18 gen 2019 alle ore 15:53 Valerio Barnabei <
> valerio.barnabei at gmail.com> ha scritto:
>
>> Hello,
>> I'm trying to figure out how to change my program in order to use MATIS
>> type and PCBDD.
>> I managed to build my MATIS matrix successfully, but my code is now
>> slower. I think the main issue is that i'm using MatSetValues inside the
>> element loop, and passing as values the e-th local element matrix. In a
>> sort of pseudocode:
>>
>> for e in localElements:{
>> for (int qp=0;qp<ngp_tot;qp++)
>>        for(int A=0;A<Nnodesloc;A++){
>>                 for(int B=0;B<Nnodesloc;B++){
>>                     //Bat dot temp (NB: ba_n0=bat_0n)
>>                     double
>> prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
>>                     //ke[A][B]+=weights[qp]*detJ[qp]*prod;
>>                     ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;
>>
>>
>> ?
>> Actually i have lots of doubts:
>> -when using ISLocalToGlobalMappingCreate(), what's the blocksize?
>>
>> Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <
>> valerio.barnabei at gmail.com> ha scritto:
>>
>>> Thank you Stefano for your quick answer, I will try to change PC as you
>>> suggest, and I will read something about MATIS type. I will let you know if
>>> I still have doubts.
>>>
>>> Valerio
>>>
>>> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <
>>> stefano.zampini at gmail.com> ha scritto:
>>>
>>>> What problem are you trying to solve?
>>>> For standard elliptic problems (e.g. poisson or elasticity) you should
>>>> use a preconditioner that scales, for example PCGAMG or BoomerAMG from
>>>> HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via
>>>> --download-hypre)
>>>> Alternatively, since you are using FEM with local Assembly, you can use
>>>> PCBDDC, but this requires few extra calls
>>>>
>>>> - call MatSetType(K,MATIS)
>>>> - provide an ISLocalToGlobalMapping of local to global degrees of
>>>> freedom before you set preallocation via MatSetLocalToGlobalMapping (if you
>>>> provide this call, then you can use MatSetValuesLocal for both AIJ
>>>> amd MATIS types)
>>>>
>>>>
>>>>
>>>> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via
>>>> petsc-users <petsc-users at mcs.anl.gov> ha scritto:
>>>>
>>>>> Hello everybody,
>>>>>
>>>>> I have some doubts about the parallel solution of a simple FEM ksp
>>>>> problem. I searched in the mailing list but the information I found are not
>>>>> helping me.
>>>>> I need to control manually the parallel assemble of the main matrix,
>>>>> but I would like to use PetSC for the solution of the final linear system
>>>>> K*U=F.
>>>>> The algorithm I am using, loads an already decomposed FEM domain, and
>>>>> each process contributes to the global matrix assembly using its portion of
>>>>> domain decomposition (made in pre-processing). In the element loop, at each
>>>>> element I load the element submatrix in the global matrix via MatSetValue
>>>>> function. Then, out of the element loop, I write
>>>>> /*
>>>>> ...matrix preallocation
>>>>> ...calculate element local stiffness matrix (local in a FEM sense and
>>>>> local in a domain decomposition sense)
>>>>> MatSetValue (ADD)
>>>>> */
>>>>> and then
>>>>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>>>>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>>>>>
>>>>> This part scales perfectly and is sufficiently fast.
>>>>> My K matrix is a sort of banded diagonal matrix: my preallocation is
>>>>> not perfect, but it works at this stage of my work.
>>>>> Finally I solve the system with the ksp logic:
>>>>>
>>>>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>>>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>>>>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>>>>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI,
>>>>> the behaviour is almost identical, the iteration number lowers but is still
>>>>> really high
>>>>>
>>>>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>>>>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>>>> ierr=KSPSolve(ksp,F,sol);
>>>>>
>>>>> When I run the code I am noticing some strange behaviours. Indeed, the
>>>>> numerical solution appear right, and the code scales if I make a simple
>>>>> strong-scaling test. However, the ksp_log and log_view (see below, logs for
>>>>> a 160k quad linear element uniform 2dmesh) shows me a huge number of
>>>>> iterations, and the time spent in "solve" function is very high.
>>>>> Furthermore, the more I increase the problem size keeping a constant load
>>>>> per process (weak scaling) the more iterations seems to be needed for
>>>>> convergence. In general, when increasing the problem size (i.e. 16M
>>>>> elements) the convergence is not achieved before the maximum iteration
>>>>> number is achieved.
>>>>>
>>>>> Question 1: Am I doing it right? Or am I making some trouble with the
>>>>> MPI communication management? (i'm aware that my domain decomposition does
>>>>> not match the petsc matrix decomposition, I'm expecting it to cripple my
>>>>> algorithm a bit)
>>>>> Question 2: Why the solver takes so much time and so many iterations?
>>>>> the problem is really simple, and the solution appears correct when
>>>>> plotted. Am I ill conditioning the system?
>>>>> Thanks in advance for your help. I can add any further information if
>>>>> needed.
>>>>>
>>>>> Valerio
>>>>>
>>>>
>>>>
>>>> --
>>>> Stefano
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/f0837f4c/attachment-0001.html>

From jed at jedbrown.org  Fri Jan 18 10:30:50 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 18 Jan 2019 09:30:50 -0700
Subject: [petsc-users] ASM Interface (Additive Schwarz Method)
In-Reply-To: <CADOhEh7DAnxeTw2fu=FKoLPMRRkwEswmSpVkZ49tg32B+gKHyA@mail.gmail.com>
References: <871s5a5cbh.fsf@uni-heidelberg.de>
	<CADOhEh7DAnxeTw2fu=FKoLPMRRkwEswmSpVkZ49tg32B+gKHyA@mail.gmail.com>
Message-ID: <871s5ac4hx.fsf@jedbrown.org>

Mark Adams via petsc-users <petsc-users at mcs.anl.gov> writes:

>> This works perfectly as long as I use PC_ASM_BASIC for the PCASMType. If I
>> switch to PC_ASM_RESTRICT, my GMRES algorithm, does not converge anymore.
>> Why is this?
>>
>
> PC_ASM_RESTRICT specifies the use of a different algorithm. It is a cheaper
> algorithm in terms of work (communication) but is weaker mathematically, in
> parallel.

Notwithstanding whatever Mark means by "weaker mathematically", RESTRICT
often/usually is observed to perform as well or better than BASIC when
using a Krylov method that can tolerate a nonsymmetric preconditioner.
Notably, BASIC does not require identifying the "local" part, just the
full (overlapping) subdomains.  So I think you might have a problem in
the definition of is_local.

If you're using one subdomain per process, the local part is typically
just the owned dofs, e.g., as returned by VecGetOwnershipRange or
MatGetOwnershipIS.  I'm having some trouble understanding your diagram
-- ownership of any rank is contiguous, but it looks like you're doing
something else.  Is that intentional?

From rickcha at googlemail.com  Fri Jan 18 11:19:46 2019
From: rickcha at googlemail.com (rickcha at googlemail.com)
Date: Fri, 18 Jan 2019 18:19:46 +0100
Subject: [petsc-users] Error in DMFieldComputeFaceData_DS with hex
 elements
In-Reply-To: <CAMYG4GkAjzNx-Rpe8QF2cdbx9dwog+vUeqQ6+Jv28H6zRnCDow@mail.gmail.com>
References: <2415BCFF-0A48-4181-8B80-8B85D62C6F21@googlemail.com>
	<CAMYG4GkAjzNx-Rpe8QF2cdbx9dwog+vUeqQ6+Jv28H6zRnCDow@mail.gmail.com>
Message-ID: <D41955A0-3B18-4D8B-ABA2-B0476D162435@googlemail.com>



> On 18. Jan 2019, at 16:09, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Jan 18, 2019 at 7:54 AM rickcha--- via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> Dear petsc-team,
> 
> I ran into an error when using petscFE in combination with a hex mesh constructed with gmsh. The error message reads:
> 
> Yes, I am guessing that GMsh put some crazy crap in the mesh file. Can you test it out on a hex mesh from DMPlexCreateBoxMesh()

I tested DMPlexCreateBoxMesh() and it throws me the same error message. I tried running example 77 with -simplex 0, but it seems that tensor product elements are not available for this example:


[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: No stored partition matching run parameters
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5  GIT Date: 2019-01-17 05:31:34 +0100
[0]PETSC ERROR: ./ex77 on a arch-linux2-intel named iket127138 by hartig Fri Jan 18 18:00:33 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel --with-debugging=false --with-cc=/home/hartig/intel/bin/icc --with-cxx=/home/hartig/intel/bin/icpc --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml --download-parmetis --download-metis --download-hypre --download-mumps --download-scalapack --download-triangle --download-hdf5 --download-ptscotch --download-chaco
[0]PETSC ERROR: #1 CreateMesh() line 443 in /home/hartig/petsc/src/snes/examples/tutorials/ex77.c
[0]PETSC ERROR: #2 main() line 594 in /home/hartig/petsc/src/snes/examples/tutorials/ex77.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -bc_fixed 1
[0]PETSC ERROR: -bc_pressure 2
[0]PETSC ERROR: -def_petscspace_degree 2
[0]PETSC ERROR: -dim 3
[0]PETSC ERROR: -dm_refine 0
[0]PETSC ERROR: -elastMat_petscspace_degree 0
[0]PETSC ERROR: -fieldsplit_deformation_ksp_type preonly
[0]PETSC ERROR: -fieldsplit_deformation_pc_type lu
[0]PETSC ERROR: -fieldsplit_pressure_ksp_rtol 1e-10
[0]PETSC ERROR: -fieldsplit_pressure_pc_type jacobi
[0]PETSC ERROR: -interpolate 1
[0]PETSC ERROR: -ksp_converged_reason
[0]PETSC ERROR: -ksp_monitor_short
[0]PETSC ERROR: -ksp_rtol 1e-10
[0]PETSC ERROR: -ksp_type fgmres
[0]PETSC ERROR: -pc_fieldsplit_schur_factorization_type upper
[0]PETSC ERROR: -pc_fieldsplit_type schur
[0]PETSC ERROR: -pc_type fieldsplit
[0]PETSC ERROR: -pres_petscspace_degree 1
[0]PETSC ERROR: -run_type full
[0]PETSC ERROR: -show_solution 0
[0]PETSC ERROR: -simplex 0
[0]PETSC ERROR: -snes_converged_reason
[0]PETSC ERROR: -snes_monitor_short
[0]PETSC ERROR: -snes_rtol 1e-05
[0]PETSC ERROR: -test_partition
[0]PETSC ERROR: -wall_pres_petscspace_degree 0
[0]PETSC ERROR: -wall_pressure 0.4
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=62
:
system msg for write_line failure : Bad file descriptor


> just ot make sure your physics works? Then we can look at the GMsh file (I hate that program).
>  
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Not implemented yet
> 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5  GIT Date: 2019-01-17 05:31:34 +0100
> [0]PETSC ERROR: ./standaloneFEM on a arch-linux2-intel-debug named iket127138 by hartig Fri Jan 18 10:49:52 2019
> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel-debug --with-debugging=true --with-cc=/home/hartig/intel/bin/icc --with-cxx=/home/hartig/intel/bin/icpc --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml --download-parmetis --download-metis --download-hypre --download-mumps --download-scalapack --download-triangle --download-hdf5 --download-ptscotch --download-chaco
> [0]PETSC ERROR: #1 DMFieldComputeFaceData_DS() line 943 in /home/hartig/petsc/src/dm/field/impls/ds/dmfieldds.c
> [0]PETSC ERROR: #2 DMFieldCreateFEGeom() line 537 in /home/hartig/petsc/src/dm/field/interface/dmfield.c
> [0]PETSC ERROR: #3 DMSNESGetFEGeom() line 2598 in /home/hartig/petsc/src/dm/impls/plex/plexfem.c
> [0]PETSC ERROR: #4 DMPlexComputeBdResidual_Single_Internal() line 1520 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #5 DMPlexComputeBdResidual_Internal() line 1649 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #6 DMPlexComputeResidual_Internal() line 1946 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
> [0]PETSC ERROR: #7 DMPlexTSComputeIFunctionFEM() line 206 in /home/hartig/petsc/src/ts/utils/dmplexts.c
> [0]PETSC ERROR: #8 ComputeI2FunctionWithInertia() line 1284 in /home/hartig/cimply/femanalysis.c
> [0]PETSC ERROR: #9 TSComputeI2Function() line 1707 in /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #10 SNESTSFormFunction_Alpha() line 339 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #11 SNESTSFormFunction() line 4694 in /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #12 SNESComputeFunction() line 2301 in /home/hartig/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 175 in /home/hartig/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #14 SNESSolve() line 4454 in /home/hartig/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #15 TSAlpha_SNESSolve() line 101 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #16 TSAlpha_Restart() line 143 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #17 TSStep_Alpha() line 217 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
> [0]PETSC ERROR: #18 TSStep() line 3664 in /home/hartig/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #19 TSSolve() line 3847 in /home/hartig/petsc/src/ts/interface/ts.c
> 
> Upon investigation, I found that petsc throws the error message because coneSize has a value of six. This is the first thing that puzzles me. I would have expected a cone size of 4 since we are dealing with faces of hex elements. Is it possible that we are on the wrong ?level? of the mesh graph somehow? Printing "numFaces? in the debugger, I get the total number of elements (not 2D faces) I have in my mesh for the hex-case.
> 
> I ran the same case with a simplex mesh and found another thing I don?t quite understand: Both times I ran with -petscspace-degree 2. Yet in case of the simplex mesh, I get a polynomial degree of 1 in DMFieldComputeFaceData_DS (?maxDegree?)
> 
> I think you are looking at the coordinate space. By default, simplex mesh use affine coordinate spaces.
>  
> and in case of the hex-mesh, I get a polynomial degree of 3.
> 
> That is strange. It should be 2.

I can?t quite follow you here. Do you refer to the local coordinate system of the element used to define the shape functions? Then a degree of 1 would mean two non-collinear vectors e_1 and e_2?

Thanks,
Max


> 
>   Thanks,
> 
>     Matt
>  
> In both cases I would have expected the degree to be 2, corresponding to Tet10 and Hex20 elements. I might be confusing two different concepts here. But this was how I understood the documentation until now.
> 
> Thanks,
> Max
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/535f6142/attachment.html>

From stefano.zampini at gmail.com  Fri Jan 18 15:18:51 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Sat, 19 Jan 2019 00:18:51 +0300
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_fAqa1DyCvU=JD8SkgyaYj-bnukpy8Pmxi+6CjWB1e4NQ@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
	<CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
	<CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>
	<CANZLC_fAqa1DyCvU=JD8SkgyaYj-bnukpy8Pmxi+6CjWB1e4NQ@mail.gmail.com>
Message-ID: <02450C04-4B5B-4603-BE0B-B61730BF6851@gmail.com>



> On Jan 18, 2019, at 6:47 PM, Valerio Barnabei <valerio.barnabei at gmail.com> wrote:
> 
> Hello (again),
> I'm trying to figure out how to change my program in order to use MATIS type and PCBDD.
> I managed to build my MATIS matrix successfully, but my code is now slower. I think the main issue is that i'm using MatSetValues inside the element loop, and passing as values the e-th local element matrix.

The only reason I may think about a slower code, is that MatSetValues for MATIS has indirect memory access when mapping global-to-local.
Are you preallocating the matrix properly? I mean, are you calling MatISSetPreallocation with the same input used for MatMPIAIJSetPreallocation?

> In a sort of pseudocode, where B is the shape functions matrix and D is a ?diffusion? matrix:
>  
> for e in localElements
> for qp in quadraturePoints{
>                         for n in localNodes {
>                                    for m in localNodes {            
>                                                ke += weights[qp]*detJ[qp]*B[n]^T* D*B[m];
> }
> }
> For n in localNodes{
> For m in localNodes{
>                 fe[n] -= ke[n* localNodes +m]*ge[e][m];//ge is a simple array that lets me easily take into account Dirichlet conditions, for my simple case
> }
> for n in localNodes{
> //            idx [n]=LM [e][n];
> //LM is the DOF array (rows are Local elements, columns are local nodes, values are global dofs, i.e. row/col indices for global assembly of K, negative values are prescribed DOFs
> }
> MatSetValues(K, localNodes, (PetscInt*) &idx [0], localNodes, (PetscInt*) &idx [0], (PetscScalar*) &ke[0], ADD_VALUES);
> //same for VecSetValues
> }
> Just to clarify: the array I used to create the map is a unique array for each process, without the negative values of the prescribed DOFs; the array I?m using in matsetvalues is each time the portion of the array used for the map relative to the current element. In addition to that: the code is working, it gives the same result of an older version where I used MATAIJ, but the portion of the element assembly loop is slower, and so is the solving process (compared using PCNONE on both, obtained same iterations to converge, same norm_2 of solution vector).
> I guess using MatSetValues outside of the loop, using the same large array I used to create the map for the matIS is supposed to be better, but I cannot see a way to use it with this elemental assembly logic.

MatSetValues (and MatSetValuesLocal) expects 2d data for the entries. Calling these once per element is the best you can do.

> Stefano suggested to use MatSetValuesLocal: in that case what is supposed to be idx? (I?m using global indices now).

You should use the corresponding local indices. I mean, if your local -> global map is (0,1,2) -> (31,44,77), then, in place of 44, you should use 1 as idx for MatSetValuesLocal

> If I use MatSetValuesLocal, can I stick with this logic of adding values inside the element loop?

Same loop, same entries. Only local indices in place of the global ones

> Is there any other way you suggest to assign values for this case?Any suggestion would be really appreciated.
> I?m not sure I explained my issue properly, I can give any further information needed.
> 
> Valerio
>  
> 
> Il giorno ven 18 gen 2019 alle ore 15:55 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
> The mail was still incomplete. Sorry, i will write it again.
> Apologies.
> 
> Il giorno ven 18 gen 2019 alle ore 15:53 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
> Hello,
> I'm trying to figure out how to change my program in order to use MATIS type and PCBDD. 
> I managed to build my MATIS matrix successfully, but my code is now slower. I think the main issue is that i'm using MatSetValues inside the element loop, and passing as values the e-th local element matrix. In a sort of pseudocode:
> 
> for e in localElements:{
> for (int qp=0;qp<ngp_tot;qp++)
>        for(int A=0;A<Nnodesloc;A++){
>                 for(int B=0;B<Nnodesloc;B++){
>                     //Bat dot temp (NB: ba_n0=bat_0n)
>                     double prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
>                     //ke[A][B]+=weights[qp]*detJ[qp]*prod;
>                     ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;
> 
> 
> ?
> Actually i have lots of doubts:
> -when using ISLocalToGlobalMappingCreate(), what's the blocksize?
> 
> Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
> Thank you Stefano for your quick answer, I will try to change PC as you suggest, and I will read something about MATIS type. I will let you know if I still have doubts.
> 
> Valerio
> 
> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <stefano.zampini at gmail.com <mailto:stefano.zampini at gmail.com>> ha scritto:
> What problem are you trying to solve?
> For standard elliptic problems (e.g. poisson or elasticity) you should use a preconditioner that scales, for example PCGAMG or BoomerAMG from HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via --download-hypre)
> Alternatively, since you are using FEM with local Assembly, you can use PCBDDC, but this requires few extra calls
> 
> - call MatSetType(K,MATIS)
> - provide an ISLocalToGlobalMapping of local to global degrees of freedom before you set preallocation via MatSetLocalToGlobalMapping (if you provide this call, then you can use MatSetValuesLocal for both AIJ amd MATIS types)
> 
> 
> 
> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> ha scritto:
> Hello everybody,
> 
> I have some doubts about the parallel solution of a simple FEM ksp problem. I searched in the mailing list but the information I found are not helping me.
> I need to control manually the parallel assemble of the main matrix, but I would like to use PetSC for the solution of the final linear system K*U=F. 
> The algorithm I am using, loads an already decomposed FEM domain, and each process contributes to the global matrix assembly using its portion of domain decomposition (made in pre-processing). In the element loop, at each element I load the element submatrix in the global matrix via MatSetValue function. Then, out of the element loop, I write
> /*
> ...matrix preallocation
> ...calculate element local stiffness matrix (local in a FEM sense and local in a domain decomposition sense)
> MatSetValue (ADD)
> */
> and then 
> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
> 
> This part scales perfectly and is sufficiently fast.
> My K matrix is a sort of banded diagonal matrix: my preallocation is not perfect, but it works at this stage of my work.
> Finally I solve the system with the ksp logic:
> 
> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI, the behaviour is almost identical, the iteration number lowers but is still really high
> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
> ierr=KSPSolve(ksp,F,sol);
> 
> When I run the code I am noticing some strange behaviours. Indeed, the numerical solution appear right, and the code scales if I make a simple strong-scaling test. However, the ksp_log and log_view (see below, logs for a 160k quad linear element uniform 2dmesh) shows me a huge number of iterations, and the time spent in "solve" function is very high. Furthermore, the more I increase the problem size keeping a constant load per process (weak scaling) the more iterations seems to be needed for convergence. In general, when increasing the problem size (i.e. 16M elements) the convergence is not achieved before the maximum iteration number is achieved. 
> 
> Question 1: Am I doing it right? Or am I making some trouble with the MPI communication management? (i'm aware that my domain decomposition does not match the petsc matrix decomposition, I'm expecting it to cripple my algorithm a bit)
> Question 2: Why the solver takes so much time and so many iterations? the problem is really simple, and the solution appears correct when plotted. Am I ill conditioning the system?
> Thanks in advance for your help. I can add any further information if needed.
> 
> Valerio
> 
> 
> -- 
> Stefano

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190119/2d503303/attachment.html>

From valerio.barnabei at gmail.com  Fri Jan 18 15:58:27 2019
From: valerio.barnabei at gmail.com (Valerio Barnabei)
Date: Fri, 18 Jan 2019 22:58:27 +0100
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <02450C04-4B5B-4603-BE0B-B61730BF6851@gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
	<CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
	<CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>
	<CANZLC_fAqa1DyCvU=JD8SkgyaYj-bnukpy8Pmxi+6CjWB1e4NQ@mail.gmail.com>
	<02450C04-4B5B-4603-BE0B-B61730BF6851@gmail.com>
Message-ID: <CANZLC_dRqxxNRLRk9Z-Rt9tzYe5EhSGJKSP-zRz6Sb_RUk9+uw@mail.gmail.com>

>
> The only reason I may think about a slower code, is that MatSetValues for
> MATIS has indirect memory access when mapping global-to-local.
>
Are you preallocating the matrix properly? I mean, are you calling
> MatISSetPreallocation with the same input used for
> MatMPIAIJSetPreallocation?
>

You're right. I'm preallocating a different number of nnz. I forgot how
much preallocation influences performance.
My doubt is: if i created my MATIS using MatCreateIS(PETSC_COMM_WORLD, 1,
PETSC_DECIDE, PETSC_DECIDE,  ndofs , ndofs, map, map, &K), how can i
preallocate *_nnz for local submatrix? Do you suggest creating MATIS by
passing the local size instead of the global size?

Same loop, same entries. Only local indices in place of the global ones
>

And the performance is supposed to be the same, right?

A couple more questions:
-I tried to use PCGAMG, and the convergence is really fast. Thank you for
your suggestion. I'm planning to add an advection term to my equation,
which preconditioner would you suggest?
-I'm also trying to use PCBDDC (as you suggested in an earlier mail), but i
got errors, basically because I'm not sure about the options i'm supposed
to use. If you think this PC could suite my case (considering the fact I
will add an advection term soon), can I get any help to tweak it and make
it work? Attached you'll find my error log of a run with -pc_type bddc. Is
there any specific requirement (that i could have completely missed) in the
domain decomposition strategy to use this particular PC?

Thank you for your help and your time.

Valerio



Il giorno ven 18 gen 2019 alle ore 22:18 Stefano Zampini <
stefano.zampini at gmail.com> ha scritto:

>
>
> On Jan 18, 2019, at 6:47 PM, Valerio Barnabei <valerio.barnabei at gmail.com>
> wrote:
>
> Hello (again),
> I'm trying to figure out how to change my program in order to use MATIS
> type and PCBDD.
> I managed to build my MATIS matrix successfully, but my code is now
> slower. I think the main issue is that i'm using MatSetValues inside the
> element loop, and passing as values the e-th local element matrix.
>
>
> The only reason I may think about a slower code, is that MatSetValues for
> MATIS has indirect memory access when mapping global-to-local.
> Are you preallocating the matrix properly? I mean, are you calling
> MatISSetPreallocation with the same input used for
> MatMPIAIJSetPreallocation?
>
> In a sort of pseudocode, where B is the shape functions matrix and D is a
> ?diffusion? matrix:
>
>
> for e in localElements
> for qp in quadraturePoints{
>                         for n in localNodes {
>                                    for m in localNodes {
>                                                ke +=
> weights[qp]*detJ[qp]*B[n]^T* D*B[m];
> }
> }
> For n in localNodes{
> For m in localNodes{
>                 fe[n] -= ke[n* localNodes +m]*ge[e][m];//ge is a simple
> array that lets me easily take into account Dirichlet conditions, for my
> simple case
> }
> for n in localNodes{
> //            idx [n]=LM [e][n];
> //LM is the DOF array (rows are Local elements, columns are local nodes,
> values are global dofs, i.e. row/col indices for global assembly of K,
> negative values are prescribed DOFs
> }
> MatSetValues(K, localNodes, (PetscInt*) &idx [0], localNodes, (PetscInt*)
> &idx [0], (PetscScalar*) &ke[0], ADD_VALUES);
> //same for VecSetValues
> }
> Just to clarify: the array I used to create the map is a unique array for
> each process, without the negative values of the prescribed DOFs; the array
> I?m using in matsetvalues is each time the portion of the array used for
> the map relative to the current element. In addition to that: the code is
> working, it gives the same result of an older version where I used MATAIJ,
> but the portion of the element assembly loop is slower, and so is the
> solving process (compared using PCNONE on both, obtained same iterations to
> converge, same norm_2 of solution vector).
> I guess using MatSetValues outside of the loop, using the same large array
> I used to create the map for the matIS is supposed to be better, but I
> cannot see a way to use it with this elemental assembly logic.
>
>
> MatSetValues (and MatSetValuesLocal) expects 2d data for the entries.
> Calling these once per element is the best you can do.
>
> Stefano suggested to use MatSetValuesLocal: in that case what is supposed
> to be idx? (I?m using global indices now).
>
>
> You should use the corresponding local indices. I mean, if your local ->
> global map is (0,1,2) -> (31,44,77), then, in place of 44, you should use 1
> as idx for MatSetValuesLocal
>
> If I use MatSetValuesLocal, can I stick with this logic of adding values
> inside the element loop?
>
>
> Same loop, same entries. Only local indices in place of the global ones
>
> Is there any other way you suggest to assign values for this case?Any
> suggestion would be really appreciated.
>
> I?m not sure I explained my issue properly, I can give any further
> information needed.
>
> Valerio
>
>
>
> Il giorno ven 18 gen 2019 alle ore 15:55 Valerio Barnabei <
> valerio.barnabei at gmail.com> ha scritto:
>
>> The mail was still incomplete. Sorry, i will write it again.
>> Apologies.
>>
>> Il giorno ven 18 gen 2019 alle ore 15:53 Valerio Barnabei <
>> valerio.barnabei at gmail.com> ha scritto:
>>
>>> Hello,
>>> I'm trying to figure out how to change my program in order to use MATIS
>>> type and PCBDD.
>>> I managed to build my MATIS matrix successfully, but my code is now
>>> slower. I think the main issue is that i'm using MatSetValues inside the
>>> element loop, and passing as values the e-th local element matrix. In a
>>> sort of pseudocode:
>>>
>>> for e in localElements:{
>>> for (int qp=0;qp<ngp_tot;qp++)
>>>        for(int A=0;A<Nnodesloc;A++){
>>>                 for(int B=0;B<Nnodesloc;B++){
>>>                     //Bat dot temp (NB: ba_n0=bat_0n)
>>>                     double
>>> prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
>>>                     //ke[A][B]+=weights[qp]*detJ[qp]*prod;
>>>                     ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;
>>>
>>>
>>> ?
>>> Actually i have lots of doubts:
>>> -when using ISLocalToGlobalMappingCreate(), what's the blocksize?
>>>
>>> Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <
>>> valerio.barnabei at gmail.com> ha scritto:
>>>
>>>> Thank you Stefano for your quick answer, I will try to change PC as you
>>>> suggest, and I will read something about MATIS type. I will let you know if
>>>> I still have doubts.
>>>>
>>>> Valerio
>>>>
>>>> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <
>>>> stefano.zampini at gmail.com> ha scritto:
>>>>
>>>>> What problem are you trying to solve?
>>>>> For standard elliptic problems (e.g. poisson or elasticity) you should
>>>>> use a preconditioner that scales, for example PCGAMG or BoomerAMG from
>>>>> HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via
>>>>> --download-hypre)
>>>>> Alternatively, since you are using FEM with local Assembly, you can
>>>>> use PCBDDC, but this requires few extra calls
>>>>>
>>>>> - call MatSetType(K,MATIS)
>>>>> - provide an ISLocalToGlobalMapping of local to global degrees of
>>>>> freedom before you set preallocation via MatSetLocalToGlobalMapping (if you
>>>>> provide this call, then you can use MatSetValuesLocal for both AIJ
>>>>> amd MATIS types)
>>>>>
>>>>>
>>>>>
>>>>> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via
>>>>> petsc-users <petsc-users at mcs.anl.gov> ha scritto:
>>>>>
>>>>>> Hello everybody,
>>>>>>
>>>>>> I have some doubts about the parallel solution of a simple FEM ksp
>>>>>> problem. I searched in the mailing list but the information I found are not
>>>>>> helping me.
>>>>>> I need to control manually the parallel assemble of the main matrix,
>>>>>> but I would like to use PetSC for the solution of the final linear system
>>>>>> K*U=F.
>>>>>> The algorithm I am using, loads an already decomposed FEM domain, and
>>>>>> each process contributes to the global matrix assembly using its portion of
>>>>>> domain decomposition (made in pre-processing). In the element loop, at each
>>>>>> element I load the element submatrix in the global matrix via MatSetValue
>>>>>> function. Then, out of the element loop, I write
>>>>>> /*
>>>>>> ...matrix preallocation
>>>>>> ...calculate element local stiffness matrix (local in a FEM sense and
>>>>>> local in a domain decomposition sense)
>>>>>> MatSetValue (ADD)
>>>>>> */
>>>>>> and then
>>>>>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>>>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>>>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>>>>>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>>>>>>
>>>>>> This part scales perfectly and is sufficiently fast.
>>>>>> My K matrix is a sort of banded diagonal matrix: my preallocation is
>>>>>> not perfect, but it works at this stage of my work.
>>>>>> Finally I solve the system with the ksp logic:
>>>>>>
>>>>>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>>>>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>>>>>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>>>>>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI,
>>>>>> the behaviour is almost identical, the iteration number lowers but is still
>>>>>> really high
>>>>>>
>>>>>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>>>>>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>>>>> ierr=KSPSolve(ksp,F,sol);
>>>>>>
>>>>>> When I run the code I am noticing some strange behaviours. Indeed,
>>>>>> the numerical solution appear right, and the code scales if I make a simple
>>>>>> strong-scaling test. However, the ksp_log and log_view (see below, logs for
>>>>>> a 160k quad linear element uniform 2dmesh) shows me a huge number of
>>>>>> iterations, and the time spent in "solve" function is very high.
>>>>>> Furthermore, the more I increase the problem size keeping a constant load
>>>>>> per process (weak scaling) the more iterations seems to be needed for
>>>>>> convergence. In general, when increasing the problem size (i.e. 16M
>>>>>> elements) the convergence is not achieved before the maximum iteration
>>>>>> number is achieved.
>>>>>>
>>>>>> Question 1: Am I doing it right? Or am I making some trouble with the
>>>>>> MPI communication management? (i'm aware that my domain decomposition does
>>>>>> not match the petsc matrix decomposition, I'm expecting it to cripple my
>>>>>> algorithm a bit)
>>>>>> Question 2: Why the solver takes so much time and so many iterations?
>>>>>> the problem is really simple, and the solution appears correct when
>>>>>> plotted. Am I ill conditioning the system?
>>>>>> Thanks in advance for your help. I can add any further information if
>>>>>> needed.
>>>>>>
>>>>>> Valerio
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Stefano
>>>>>
>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/02a8c712/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.out
Type: application/octet-stream
Size: 122246 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190118/02a8c712/attachment-0001.obj>

From stefano.zampini at gmail.com  Mon Jan 21 14:02:50 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 21 Jan 2019 23:02:50 +0300
Subject: [petsc-users] KSP solve time and iterations
In-Reply-To: <CANZLC_dRqxxNRLRk9Z-Rt9tzYe5EhSGJKSP-zRz6Sb_RUk9+uw@mail.gmail.com>
References: <CANZLC_fp7P0Tno07kvc-4zp4EibQ-u1cew2KBSfCs+f2REN05A@mail.gmail.com>
	<CAGPUishfosa4bxo6mMQaDHMs6_E18pzkt7=uy=ZDPdheV8aYVw@mail.gmail.com>
	<CANZLC_fdJ99Kz1n4o6KqnZxvu_qv+zk_5OmCSaivY5rswGhNAQ@mail.gmail.com>
	<CANZLC_f5QBXvU2MG2yRjgK7TNmAggAYAOWXAFvqkc0+vw6UxBw@mail.gmail.com>
	<CANZLC_fsboh4Y==cssZUBH0VUJN_A4nkC1apcJwcxWLjrPFhqA@mail.gmail.com>
	<CANZLC_fAqa1DyCvU=JD8SkgyaYj-bnukpy8Pmxi+6CjWB1e4NQ@mail.gmail.com>
	<02450C04-4B5B-4603-BE0B-B61730BF6851@gmail.com>
	<CANZLC_dRqxxNRLRk9Z-Rt9tzYe5EhSGJKSP-zRz6Sb_RUk9+uw@mail.gmail.com>
Message-ID: <DC016CB5-FC2F-422A-B0FD-605FFDBBAAA1@gmail.com>



> On Jan 19, 2019, at 12:58 AM, Valerio Barnabei <valerio.barnabei at gmail.com> wrote:
> 
> The only reason I may think about a slower code, is that MatSetValues for MATIS has indirect memory access when mapping global-to-local.
> Are you preallocating the matrix properly? I mean, are you calling MatISSetPreallocation with the same input used for MatMPIAIJSetPreallocation?
>  
> You're right. I'm preallocating a different number of nnz. I forgot how much preallocation influences performance. 
> My doubt is: if i created my MATIS using MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE,  ndofs , ndofs, map, map, &K), how can i preallocate *_nnz for local submatrix? Do you suggest creating MATIS by passing the local size instead of the global size?

The local size is not the size of the local FEM problems. It is the size of the local part of the global vector used in MatMult operations
If you want to do preallocation on the local FEM matrices, you do

MatCreateIS(?&A)
MatISGetLocalMat(A,&B)
MatSeqAIJSetPreallocation(B,local_nonzeros?)

> 
> Same loop, same entries. Only local indices in place of the global ones
> 
> And the performance is supposed to be the same, right?

Same. Quoting Donald Knuth: Premature optimization is the root of all evil
> 
> A couple more questions:
> -I tried to use PCGAMG, and the convergence is really fast. Thank you for your suggestion. I'm planning to add an advection term to my equation, which preconditioner would you suggest?

Fast multigrid convergence with strong advecting terms is always tricky.

> -I'm also trying to use PCBDDC (as you suggested in an earlier mail), but i got errors, basically because I'm not sure about the options i'm supposed to use. If you think this PC could suite my case (considering the fact I will add an advection term soon), can I get any help to tweak it and make it work? Attached you'll find my error log of a run with -pc_type bddc. Is there any specific requirement (that i could have completely missed) in the domain decomposition strategy to use this particular PC?

From the log, I see a bunch of errors coming from MatSetValues. The local-to-global map you have created is wrong.
I have an old branch where I create the l2g map brute force by doing a first pass of the values insertion loop (without actually inserting) and and then taking a unique list of global indices visited. It is not the most recommended way to create the map, but at least you have a comparison to understand where your code is wrong
https://bitbucket.org/petsc/petsc/src/e1fa67b2d58bc4882f68a386d1578f90057aefbb/src/ksp/ksp/examples/tutorials/ex56.c?at=stefano_zampini/feature-pcgamg-useamat#lines-125 <https://bitbucket.org/petsc/petsc/src/e1fa67b2d58bc4882f68a386d1578f90057aefbb/src/ksp/ksp/examples/tutorials/ex56.c?at=stefano_zampini/feature-pcgamg-useamat#lines-125>

The only error with BDDC is not actually from BDDC, but from the local solvers, since your local matrices have a zero on the diagonal, and the default LU solver from PETSc does not tolerate this (no pivoting).
> 
> Thank you for your help and your time.
> 
> Valerio
> 
> 
> 
> Il giorno ven 18 gen 2019 alle ore 22:18 Stefano Zampini <stefano.zampini at gmail.com <mailto:stefano.zampini at gmail.com>> ha scritto:
> 
> 
>> On Jan 18, 2019, at 6:47 PM, Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> wrote:
>> 
>> Hello (again),
>> I'm trying to figure out how to change my program in order to use MATIS type and PCBDD.
>> I managed to build my MATIS matrix successfully, but my code is now slower. I think the main issue is that i'm using MatSetValues inside the element loop, and passing as values the e-th local element matrix.
> 
> The only reason I may think about a slower code, is that MatSetValues for MATIS has indirect memory access when mapping global-to-local.
> Are you preallocating the matrix properly? I mean, are you calling MatISSetPreallocation with the same input used for MatMPIAIJSetPreallocation?
> 
>> In a sort of pseudocode, where B is the shape functions matrix and D is a ?diffusion? matrix:
>>  
>> for e in localElements
>> for qp in quadraturePoints{
>>                         for n in localNodes {
>>                                    for m in localNodes {            
>>                                                ke += weights[qp]*detJ[qp]*B[n]^T* D*B[m];
>> }
>> }
>> For n in localNodes{
>> For m in localNodes{
>>                 fe[n] -= ke[n* localNodes +m]*ge[e][m];//ge is a simple array that lets me easily take into account Dirichlet conditions, for my simple case
>> }
>> for n in localNodes{
>> //            idx [n]=LM [e][n];
>> //LM is the DOF array (rows are Local elements, columns are local nodes, values are global dofs, i.e. row/col indices for global assembly of K, negative values are prescribed DOFs
>> }
>> MatSetValues(K, localNodes, (PetscInt*) &idx [0], localNodes, (PetscInt*) &idx [0], (PetscScalar*) &ke[0], ADD_VALUES);
>> //same for VecSetValues
>> }
>> Just to clarify: the array I used to create the map is a unique array for each process, without the negative values of the prescribed DOFs; the array I?m using in matsetvalues is each time the portion of the array used for the map relative to the current element. In addition to that: the code is working, it gives the same result of an older version where I used MATAIJ, but the portion of the element assembly loop is slower, and so is the solving process (compared using PCNONE on both, obtained same iterations to converge, same norm_2 of solution vector).
>> I guess using MatSetValues outside of the loop, using the same large array I used to create the map for the matIS is supposed to be better, but I cannot see a way to use it with this elemental assembly logic.
> 
> MatSetValues (and MatSetValuesLocal) expects 2d data for the entries. Calling these once per element is the best you can do.
> 
>> Stefano suggested to use MatSetValuesLocal: in that case what is supposed to be idx? (I?m using global indices now).
> 
> You should use the corresponding local indices. I mean, if your local -> global map is (0,1,2) -> (31,44,77), then, in place of 44, you should use 1 as idx for MatSetValuesLocal
> 
>> If I use MatSetValuesLocal, can I stick with this logic of adding values inside the element loop?
> 
> Same loop, same entries. Only local indices in place of the global ones
> 
>> Is there any other way you suggest to assign values for this case?Any suggestion would be really appreciated.
>> I?m not sure I explained my issue properly, I can give any further information needed.
>> 
>> Valerio
>>  
>> 
>> Il giorno ven 18 gen 2019 alle ore 15:55 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
>> The mail was still incomplete. Sorry, i will write it again.
>> Apologies.
>> 
>> Il giorno ven 18 gen 2019 alle ore 15:53 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
>> Hello,
>> I'm trying to figure out how to change my program in order to use MATIS type and PCBDD. 
>> I managed to build my MATIS matrix successfully, but my code is now slower. I think the main issue is that i'm using MatSetValues inside the element loop, and passing as values the e-th local element matrix. In a sort of pseudocode:
>> 
>> for e in localElements:{
>> for (int qp=0;qp<ngp_tot;qp++)
>>        for(int A=0;A<Nnodesloc;A++){
>>                 for(int B=0;B<Nnodesloc;B++){
>>                     //Bat dot temp (NB: ba_n0=bat_0n)
>>                     double prod=temp[0][A][qp]*BaT[B][0][qp]+temp[1][A][qp]*BaT[B][1][qp];
>>                     //ke[A][B]+=weights[qp]*detJ[qp]*prod;
>>                     ke[A*Nnodesloc+B]+=weights[qp]*detJ[qp]*prod;
>> 
>> 
>> ?
>> Actually i have lots of doubts:
>> -when using ISLocalToGlobalMappingCreate(), what's the blocksize?
>> 
>> Il giorno lun 14 gen 2019 alle ore 15:08 Valerio Barnabei <valerio.barnabei at gmail.com <mailto:valerio.barnabei at gmail.com>> ha scritto:
>> Thank you Stefano for your quick answer, I will try to change PC as you suggest, and I will read something about MATIS type. I will let you know if I still have doubts.
>> 
>> Valerio
>> 
>> Il giorno dom 13 gen 2019 alle ore 16:08 Stefano Zampini <stefano.zampini at gmail.com <mailto:stefano.zampini at gmail.com>> ha scritto:
>> What problem are you trying to solve?
>> For standard elliptic problems (e.g. poisson or elasticity) you should use a preconditioner that scales, for example PCGAMG or BoomerAMG from HYPRE (PCHYPRE, if you configured PETSc with support for HYPRE via --download-hypre)
>> Alternatively, since you are using FEM with local Assembly, you can use PCBDDC, but this requires few extra calls
>> 
>> - call MatSetType(K,MATIS)
>> - provide an ISLocalToGlobalMapping of local to global degrees of freedom before you set preallocation via MatSetLocalToGlobalMapping (if you provide this call, then you can use MatSetValuesLocal for both AIJ amd MATIS types)
>> 
>> 
>> 
>> Il giorno dom 13 gen 2019 alle ore 17:59 Valerio Barnabei via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> ha scritto:
>> Hello everybody,
>> 
>> I have some doubts about the parallel solution of a simple FEM ksp problem. I searched in the mailing list but the information I found are not helping me.
>> I need to control manually the parallel assemble of the main matrix, but I would like to use PetSC for the solution of the final linear system K*U=F. 
>> The algorithm I am using, loads an already decomposed FEM domain, and each process contributes to the global matrix assembly using its portion of domain decomposition (made in pre-processing). In the element loop, at each element I load the element submatrix in the global matrix via MatSetValue function. Then, out of the element loop, I write
>> /*
>> ...matrix preallocation
>> ...calculate element local stiffness matrix (local in a FEM sense and local in a domain decomposition sense)
>> MatSetValue (ADD)
>> */
>> and then 
>> ierr=MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> ierr=MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> ierr=VecAssemblyBegin(F);CHKERRQ(ierr);
>> ierr=VecAssemblyEnd(F);CHKERRQ(ierr);
>> 
>> This part scales perfectly and is sufficiently fast.
>> My K matrix is a sort of banded diagonal matrix: my preallocation is not perfect, but it works at this stage of my work.
>> Finally I solve the system with the ksp logic:
>> 
>> ierr=KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>> ierr=KSPSetOperators(ksp,K,K);CHKERRQ(ierr);
>> ierr=KSPGetPC(ksp,&pc);CHKERRQ(ierr);
>> ierr=PCSetType(pc,PCBJACOBI);CHKERRQ(ierr); //or PCNONE, or PCJACOBI, the behaviour is almost identical, the iteration number lowers but is still really high
>> ierr=KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
>> ierr=KSPSetFromOptions(ksp);CHKERRQ(ierr);
>> ierr=KSPSolve(ksp,F,sol);
>> 
>> When I run the code I am noticing some strange behaviours. Indeed, the numerical solution appear right, and the code scales if I make a simple strong-scaling test. However, the ksp_log and log_view (see below, logs for a 160k quad linear element uniform 2dmesh) shows me a huge number of iterations, and the time spent in "solve" function is very high. Furthermore, the more I increase the problem size keeping a constant load per process (weak scaling) the more iterations seems to be needed for convergence. In general, when increasing the problem size (i.e. 16M elements) the convergence is not achieved before the maximum iteration number is achieved. 
>> 
>> Question 1: Am I doing it right? Or am I making some trouble with the MPI communication management? (i'm aware that my domain decomposition does not match the petsc matrix decomposition, I'm expecting it to cripple my algorithm a bit)
>> Question 2: Why the solver takes so much time and so many iterations? the problem is really simple, and the solution appears correct when plotted. Am I ill conditioning the system?
>> Thanks in advance for your help. I can add any further information if needed.
>> 
>> Valerio
>> 
>> 
>> -- 
>> Stefano
> 
> <log.out>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190121/e46dd676/attachment-0001.html>

From benoit.nennig at supmeca.fr  Tue Jan 22 03:57:21 2019
From: benoit.nennig at supmeca.fr (NENNIG Benoit)
Date: Tue, 22 Jan 2019 09:57:21 +0000
Subject: [petsc-users] Create 2 matrix with the same OwnershipRanges
Message-ID: <1548151039123.29819@supmeca.fr>

Hi,

I would like to create a matrix aij with the same OwnershipRanges as an other existing matrix with petsc4py. The two matrices have not the same non zeros patern. 

Any ideas ?


Regards,

Benoit

From rickcha at googlemail.com  Tue Jan 22 04:18:44 2019
From: rickcha at googlemail.com (rickcha at googlemail.com)
Date: Tue, 22 Jan 2019 11:18:44 +0100
Subject: [petsc-users] Error in DMFieldComputeFaceData_DS with hex
 elements
In-Reply-To: <D41955A0-3B18-4D8B-ABA2-B0476D162435@googlemail.com>
References: <2415BCFF-0A48-4181-8B80-8B85D62C6F21@googlemail.com>
	<CAMYG4GkAjzNx-Rpe8QF2cdbx9dwog+vUeqQ6+Jv28H6zRnCDow@mail.gmail.com>
	<D41955A0-3B18-4D8B-ABA2-B0476D162435@googlemail.com>
Message-ID: <A2417319-BF42-40FB-A20E-682BCD0FE82E@googlemail.com>



> On 18. Jan 2019, at 18:19, rickcha at googlemail.com wrote:
> 
> 
> 
>> On 18. Jan 2019, at 16:09, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Fri, Jan 18, 2019 at 7:54 AM rickcha--- via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> Dear petsc-team,
>> 
>> I ran into an error when using petscFE in combination with a hex mesh constructed with gmsh. The error message reads:
>> 
>> Yes, I am guessing that GMsh put some crazy crap in the mesh file. Can you test it out on a hex mesh from DMPlexCreateBoxMesh()
> 
> I tested DMPlexCreateBoxMesh() and it throws me the same error message. I tried running example 77 with -simplex 0, but it seems that tensor product elements are not available for this example:
> 
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: No stored partition matching run parameters
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5  GIT Date: 2019-01-17 05:31:34 +0100
> [0]PETSC ERROR: ./ex77 on a arch-linux2-intel named iket127138 by hartig Fri Jan 18 18:00:33 2019
> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel --with-debugging=false --with-cc=/home/hartig/intel/bin/icc --with-cxx=/home/hartig/intel/bin/icpc --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml --download-parmetis --download-metis --download-hypre --download-mumps --download-scalapack --download-triangle --download-hdf5 --download-ptscotch --download-chaco
> [0]PETSC ERROR: #1 CreateMesh() line 443 in /home/hartig/petsc/src/snes/examples/tutorials/ex77.c
> [0]PETSC ERROR: #2 main() line 594 in /home/hartig/petsc/src/snes/examples/tutorials/ex77.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -bc_fixed 1
> [0]PETSC ERROR: -bc_pressure 2
> [0]PETSC ERROR: -def_petscspace_degree 2
> [0]PETSC ERROR: -dim 3
> [0]PETSC ERROR: -dm_refine 0
> [0]PETSC ERROR: -elastMat_petscspace_degree 0
> [0]PETSC ERROR: -fieldsplit_deformation_ksp_type preonly
> [0]PETSC ERROR: -fieldsplit_deformation_pc_type lu
> [0]PETSC ERROR: -fieldsplit_pressure_ksp_rtol 1e-10
> [0]PETSC ERROR: -fieldsplit_pressure_pc_type jacobi
> [0]PETSC ERROR: -interpolate 1
> [0]PETSC ERROR: -ksp_converged_reason
> [0]PETSC ERROR: -ksp_monitor_short
> [0]PETSC ERROR: -ksp_rtol 1e-10
> [0]PETSC ERROR: -ksp_type fgmres
> [0]PETSC ERROR: -pc_fieldsplit_schur_factorization_type upper
> [0]PETSC ERROR: -pc_fieldsplit_type schur
> [0]PETSC ERROR: -pc_type fieldsplit
> [0]PETSC ERROR: -pres_petscspace_degree 1
> [0]PETSC ERROR: -run_type full
> [0]PETSC ERROR: -show_solution 0
> [0]PETSC ERROR: -simplex 0
> [0]PETSC ERROR: -snes_converged_reason
> [0]PETSC ERROR: -snes_monitor_short
> [0]PETSC ERROR: -snes_rtol 1e-05
> [0]PETSC ERROR: -test_partition
> [0]PETSC ERROR: -wall_pres_petscspace_degree 0
> [0]PETSC ERROR: -wall_pressure 0.4
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=62
> :
> system msg for write_line failure : Bad file descriptor
> 
> 
>> just ot make sure your physics works? Then we can look at the GMsh file (I hate that program).
>>  
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Not implemented yet
>> 
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1227-gcd612eedf5  GIT Date: 2019-01-17 05:31:34 +0100
>> [0]PETSC ERROR: ./standaloneFEM on a arch-linux2-intel-debug named iket127138 by hartig Fri Jan 18 10:49:52 2019
>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-intel-debug --with-debugging=true --with-cc=/home/hartig/intel/bin/icc --with-cxx=/home/hartig/intel/bin/icpc --with-fc=/home/hartig/intel/bin/ifort --download-mpich --download-ml --download-parmetis --download-metis --download-hypre --download-mumps --download-scalapack --download-triangle --download-hdf5 --download-ptscotch --download-chaco
>> [0]PETSC ERROR: #1 DMFieldComputeFaceData_DS() line 943 in /home/hartig/petsc/src/dm/field/impls/ds/dmfieldds.c
>> [0]PETSC ERROR: #2 DMFieldCreateFEGeom() line 537 in /home/hartig/petsc/src/dm/field/interface/dmfield.c
>> [0]PETSC ERROR: #3 DMSNESGetFEGeom() line 2598 in /home/hartig/petsc/src/dm/impls/plex/plexfem.c
>> [0]PETSC ERROR: #4 DMPlexComputeBdResidual_Single_Internal() line 1520 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
>> [0]PETSC ERROR: #5 DMPlexComputeBdResidual_Internal() line 1649 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
>> [0]PETSC ERROR: #6 DMPlexComputeResidual_Internal() line 1946 in /home/hartig/petsc/src/snes/utils/dmplexsnes.c
>> [0]PETSC ERROR: #7 DMPlexTSComputeIFunctionFEM() line 206 in /home/hartig/petsc/src/ts/utils/dmplexts.c
>> [0]PETSC ERROR: #8 ComputeI2FunctionWithInertia() line 1284 in /home/hartig/cimply/femanalysis.c
>> [0]PETSC ERROR: #9 TSComputeI2Function() line 1707 in /home/hartig/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: #10 SNESTSFormFunction_Alpha() line 339 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
>> [0]PETSC ERROR: #11 SNESTSFormFunction() line 4694 in /home/hartig/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: #12 SNESComputeFunction() line 2301 in /home/hartig/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 175 in /home/hartig/petsc/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #14 SNESSolve() line 4454 in /home/hartig/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: #15 TSAlpha_SNESSolve() line 101 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
>> [0]PETSC ERROR: #16 TSAlpha_Restart() line 143 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
>> [0]PETSC ERROR: #17 TSStep_Alpha() line 217 in /home/hartig/petsc/src/ts/impls/implicit/alpha/alpha2.c
>> [0]PETSC ERROR: #18 TSStep() line 3664 in /home/hartig/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: #19 TSSolve() line 3847 in /home/hartig/petsc/src/ts/interface/ts.c
>> 
>> Upon investigation, I found that petsc throws the error message because coneSize has a value of six. This is the first thing that puzzles me. I would have expected a cone size of 4 since we are dealing with faces of hex elements. Is it possible that we are on the wrong ?level? of the mesh graph somehow? Printing "numFaces? in the debugger, I get the total number of elements (not 2D faces) I have in my mesh for the hex-case.
>> 
>> I ran the same case with a simplex mesh and found another thing I don?t quite understand: Both times I ran with -petscspace-degree 2. Yet in case of the simplex mesh, I get a polynomial degree of 1 in DMFieldComputeFaceData_DS (?maxDegree?)
>> 
>> I think you are looking at the coordinate space. By default, simplex mesh use affine coordinate spaces.
>>  
>> and in case of the hex-mesh, I get a polynomial degree of 3.
>> 
>> That is strange. It should be 2.
> 
> I can?t quite follow you here. Do you refer to the local coordinate system of the element used to define the shape functions? Then a degree of 1 would mean two non-collinear vectors e_1 and e_2?
> 
> Thanks,
> Max
> 
> 
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> In both cases I would have expected the degree to be 2, corresponding to Tet10 and Hex20 elements. I might be confusing two different concepts here. But this was how I understood the documentation until now.
>> 
>> Thanks,
>> Max
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/


I believe I have found the issue in dmfieldds.c: 
While the first part of the code (starting line 785 in file dmfieldds.c) references the cell discretisation, the second part of the code references the face discretisation. However, the reference cell / reference DM (variable ?K? in the code) is never updated. The sorting of the quadrature points seems correct to me. But instead of a coneSize of 4 - as one would expect for a quad face - we get a coneSize of 6 because the reference DM is still the hex cell.
Hope this makes sense.

The following modifications made the code run correctly (as far as I?ve tested it) but are quite a hack:


@@ -908,7 +908,7 @@ static PetscErrorCode DMFieldComputeFaceData_DS(DMField field, IS pointIS, Petsc
               orientPoints[o][2 * q + 1] = -(lambdao[0] + lambdao[1]) + lambdao[2];
             }
             break;
-          case 4:
+          case 6:
             for (q = 0; q < Nq; q++) {
               PetscReal xi[2], xio[2];
               PetscInt oabs = (orient >= 0) ? orient : -(orient + 1);
@@ -942,6 +942,7 @@ static PetscErrorCode DMFieldComputeFaceData_DS(DMField field, IS pointIS, Petsc
           default:
             SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Not implemented yet\n");
           }
+         break;
         default:
           SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Not implemented yet\n");
         }


Unfortunately, my understanding is too limited to fix this the proper way. Probably one should swap the reference DM from cell to face somewhere along the way, but I?m not sure where.


Thanks,
Max


From knepley at gmail.com  Tue Jan 22 08:15:22 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Jan 2019 09:15:22 -0500
Subject: [petsc-users] Create 2 matrix with the same OwnershipRanges
In-Reply-To: <1548151039123.29819@supmeca.fr>
References: <1548151039123.29819@supmeca.fr>
Message-ID: <CAMYG4G=X6zrCgzqwAfJCXk917ZetU35U72scvbia_NMpbyVEOA@mail.gmail.com>

On Tue, Jan 22, 2019 at 4:57 AM NENNIG Benoit via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I would like to create a matrix aij with the same OwnershipRanges as an
> other existing matrix with petsc4py. The two matrices have not the same non
> zeros patern.
>
> Any ideas ?
>

MatSetSizes() can be used to determine the parallel layout of a matrix. So
you could call MatGetLocalSize() on the first Mat,
and then MatSetSizes() on the copy.

  Thanks,

    Matt


>
> Regards,
>
> Benoit
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/e66a0988/attachment.html>

From maahi.buet at gmail.com  Tue Jan 22 15:15:00 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Tue, 22 Jan 2019 16:15:00 -0500
Subject: [petsc-users] Accessing the elements of a Matrix
Message-ID: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>

Hello All,

I was trying to use ' MatGetArrayF90' to access a matrix and check its
values. But I couldn't do it, probably because of the specific compiler I
am using.
Is there any other way to access a matrix and check its values ?


Regards,
Maahi Talukder
Clarkson University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/f68bd72b/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan 22 15:30:02 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 22 Jan 2019 21:30:02 +0000
Subject: [petsc-users] Accessing the elements of a Matrix
In-Reply-To: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>
References: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>
Message-ID: <20BBAEAE-B44E-4FDC-977B-9A892857226D@anl.gov>


   MatGetArrayF90 is an out-dated routine. For dense matrices you can use MatDenseGetArrayF90()

  For sparse matrices you can use MatGetValues() or MatGetRow()

   Barry


> On Jan 22, 2019, at 3:15 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello All,
> 
> I was trying to use ' MatGetArrayF90' to access a matrix and check its values. But I couldn't do it, probably because of the specific compiler I am using. 
> Is there any other way to access a matrix and check its values ?
> 
> 
> Regards,
> Maahi Talukder
> Clarkson University


From fdkong.jd at gmail.com  Tue Jan 22 15:29:38 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 22 Jan 2019 14:29:38 -0700
Subject: [petsc-users] Accessing the elements of a Matrix
In-Reply-To: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>
References: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>
Message-ID: <CAN5Wd-JdyJjqXK3N3RNmG=Tp4eiv2HnJP8SNKP1JSSR8zQP2Pg@mail.gmail.com>

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html

On Tue, Jan 22, 2019 at 2:16 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello All,
>
> I was trying to use ' MatGetArrayF90' to access a matrix and check its
> values. But I couldn't do it, probably because of the specific compiler I
> am using.
> Is there any other way to access a matrix and check its values ?
>
>
> Regards,
> Maahi Talukder
> Clarkson University
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/6b923666/attachment.html>

From jychang48 at gmail.com  Tue Jan 22 16:00:39 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 22 Jan 2019 15:00:39 -0700
Subject: [petsc-users] Help with petsc4py and PC Python type
Message-ID: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>

Hi all,

I am attempting to write my own scour preconditioner for a FEniCS/petsc4py.
I managed to attach a DMShell to the KSP so that I can employ command-line
options for field splitting. Suppose I want to solve the Navier-Stokes
equation and have the following command-line options:

-ksp_type fgmres
-pc_type fieldsplit
-pc_fieldsplit_type schur
-pc_fieldsplit_schur_fact_type full
-fieldsplit_0_ksp_type gmres
-fieldsplit_0_pc_type hypre
-fieldsplit_1_ksp_type preonly
-fieldsplit_1_pc_type python
-fieldsplit_1_pc_python_type pfibs.MyPCD

Where MyPCD is part of my pfibs module and looks something like this:

class MyPCD(object):
    def setUp(self, pc):
          # Do something #
             ....
    def apply(self, pc, x, y):
          # Do something #
             ....

The problem is that in my actual python code (mycode.py) I have a dict()
object that I would like to pass into the setUp() call inside MyPCD to read
in, but it is not clear how I can do this if I set the pc python type via
command line options. If I hard-coded this, I would just call
pc.setPythonContext(...) but I was wondering if there's a way to do this,
e.g., attaching an appctx/dict to the pc itself. Does anyone have ideas or
suggestions on how to go about doing this?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/7a5e35a3/attachment.html>

From jed at jedbrown.org  Tue Jan 22 16:27:34 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 22 Jan 2019 15:27:34 -0700
Subject: [petsc-users] Help with petsc4py and PC Python type
In-Reply-To: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>
References: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>
Message-ID: <875zug2uqx.fsf@jedbrown.org>

Justin Chang via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi all,
>
> I am attempting to write my own scour preconditioner for a FEniCS/petsc4py.
> I managed to attach a DMShell to the KSP so that I can employ command-line
> options for field splitting. Suppose I want to solve the Navier-Stokes
> equation and have the following command-line options:
>
> -ksp_type fgmres
> -pc_type fieldsplit
> -pc_fieldsplit_type schur
> -pc_fieldsplit_schur_fact_type full
> -fieldsplit_0_ksp_type gmres
> -fieldsplit_0_pc_type hypre
> -fieldsplit_1_ksp_type preonly
> -fieldsplit_1_pc_type python
> -fieldsplit_1_pc_python_type pfibs.MyPCD
>
> Where MyPCD is part of my pfibs module and looks something like this:
>
> class MyPCD(object):
>     def setUp(self, pc):
>           # Do something #
>              ....
>     def apply(self, pc, x, y):
>           # Do something #
>              ....
>
> The problem is that in my actual python code (mycode.py) I have a dict()

Where does this dict come from?  When is it available?

> object that I would like to pass into the setUp() call inside MyPCD to read
> in, but it is not clear how I can do this if I set the pc python type via
> command line options. If I hard-coded this, I would just call
> pc.setPythonContext(...) but I was wondering if there's a way to do this,
> e.g., attaching an appctx/dict to the pc itself. Does anyone have ideas or
> suggestions on how to go about doing this?
>
> Thanks,
> Justin

From jychang48 at gmail.com  Tue Jan 22 16:30:30 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 22 Jan 2019 15:30:30 -0700
Subject: [petsc-users] Help with petsc4py and PC Python type
In-Reply-To: <875zug2uqx.fsf@jedbrown.org>
References: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>
	<875zug2uqx.fsf@jedbrown.org>
Message-ID: <CAP2=TMgTRmXnAmLieeeQ5QjysPk=jCo6r5Go2+KdARuukokLDA@mail.gmail.com>

As of right now, the dict() is something the user constructs inside
mycode.py. It would contain things like the viscosity, boundary conditions,
function space, etc which are needed to construct the PC operators in
MyPCD. I believe I saw the Firedrake guys do something similar to this

On Tue, Jan 22, 2019 at 3:27 PM Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Hi all,
> >
> > I am attempting to write my own scour preconditioner for a
> FEniCS/petsc4py.
> > I managed to attach a DMShell to the KSP so that I can employ
> command-line
> > options for field splitting. Suppose I want to solve the Navier-Stokes
> > equation and have the following command-line options:
> >
> > -ksp_type fgmres
> > -pc_type fieldsplit
> > -pc_fieldsplit_type schur
> > -pc_fieldsplit_schur_fact_type full
> > -fieldsplit_0_ksp_type gmres
> > -fieldsplit_0_pc_type hypre
> > -fieldsplit_1_ksp_type preonly
> > -fieldsplit_1_pc_type python
> > -fieldsplit_1_pc_python_type pfibs.MyPCD
> >
> > Where MyPCD is part of my pfibs module and looks something like this:
> >
> > class MyPCD(object):
> >     def setUp(self, pc):
> >           # Do something #
> >              ....
> >     def apply(self, pc, x, y):
> >           # Do something #
> >              ....
> >
> > The problem is that in my actual python code (mycode.py) I have a dict()
>
> Where does this dict come from?  When is it available?
>
> > object that I would like to pass into the setUp() call inside MyPCD to
> read
> > in, but it is not clear how I can do this if I set the pc python type via
> > command line options. If I hard-coded this, I would just call
> > pc.setPythonContext(...) but I was wondering if there's a way to do this,
> > e.g., attaching an appctx/dict to the pc itself. Does anyone have ideas
> or
> > suggestions on how to go about doing this?
> >
> > Thanks,
> > Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/c8df83ce/attachment.html>

From wence at gmx.li  Tue Jan 22 16:49:11 2019
From: wence at gmx.li (Lawrence Mitchell)
Date: Tue, 22 Jan 2019 22:49:11 +0000
Subject: [petsc-users] Help with petsc4py and PC Python type
In-Reply-To: <CAP2=TMgTRmXnAmLieeeQ5QjysPk=jCo6r5Go2+KdARuukokLDA@mail.gmail.com>
References: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>
	<875zug2uqx.fsf@jedbrown.org>
	<CAP2=TMgTRmXnAmLieeeQ5QjysPk=jCo6r5Go2+KdARuukokLDA@mail.gmail.com>
Message-ID: <997B4EFE-8879-4923-9ED9-19860333E92A@gmx.li>



> On 22 Jan 2019, at 22:30, Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> As of right now, the dict() is something the user constructs inside mycode.py. It would contain things like the viscosity, boundary conditions, function space, etc which are needed to construct the PC operators in MyPCD. I believe I saw the Firedrake guys do something similar to this

I attach it as an application context to the DM and make sure it is transferred when I split the dm into pieces.

Then you do pc.getDM().getAppCtx() or similar. 

Lawrence 

From jed at jedbrown.org  Tue Jan 22 16:52:11 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 22 Jan 2019 15:52:11 -0700
Subject: [petsc-users] Help with petsc4py and PC Python type
In-Reply-To: <CAP2=TMgTRmXnAmLieeeQ5QjysPk=jCo6r5Go2+KdARuukokLDA@mail.gmail.com>
References: <CAP2=TMjF8XSGDRRX9FNVAJKhzinomh7=jKdC+6w5SgCa5XPZEA@mail.gmail.com>
	<875zug2uqx.fsf@jedbrown.org>
	<CAP2=TMgTRmXnAmLieeeQ5QjysPk=jCo6r5Go2+KdARuukokLDA@mail.gmail.com>
Message-ID: <87womw1f1g.fsf@jedbrown.org>

My preference is to put this auxiliary information on the Mat (or the
Mat's DM; see MatGetDM()).  It's really declarative information about
the operator, not algorithmic parameters for the PC (like a threshold in
AMG, number of smoothing iterations, etc.).

Justin Chang <jychang48 at gmail.com> writes:

> As of right now, the dict() is something the user constructs inside
> mycode.py. It would contain things like the viscosity, boundary conditions,
> function space, etc which are needed to construct the PC operators in
> MyPCD. I believe I saw the Firedrake guys do something similar to this
>
> On Tue, Jan 22, 2019 at 3:27 PM Jed Brown <jed at jedbrown.org> wrote:
>
>> Justin Chang via petsc-users <petsc-users at mcs.anl.gov> writes:
>>
>> > Hi all,
>> >
>> > I am attempting to write my own scour preconditioner for a
>> FEniCS/petsc4py.
>> > I managed to attach a DMShell to the KSP so that I can employ
>> command-line
>> > options for field splitting. Suppose I want to solve the Navier-Stokes
>> > equation and have the following command-line options:
>> >
>> > -ksp_type fgmres
>> > -pc_type fieldsplit
>> > -pc_fieldsplit_type schur
>> > -pc_fieldsplit_schur_fact_type full
>> > -fieldsplit_0_ksp_type gmres
>> > -fieldsplit_0_pc_type hypre
>> > -fieldsplit_1_ksp_type preonly
>> > -fieldsplit_1_pc_type python
>> > -fieldsplit_1_pc_python_type pfibs.MyPCD
>> >
>> > Where MyPCD is part of my pfibs module and looks something like this:
>> >
>> > class MyPCD(object):
>> >     def setUp(self, pc):
>> >           # Do something #
>> >              ....
>> >     def apply(self, pc, x, y):
>> >           # Do something #
>> >              ....
>> >
>> > The problem is that in my actual python code (mycode.py) I have a dict()
>>
>> Where does this dict come from?  When is it available?
>>
>> > object that I would like to pass into the setUp() call inside MyPCD to
>> read
>> > in, but it is not clear how I can do this if I set the pc python type via
>> > command line options. If I hard-coded this, I would just call
>> > pc.setPythonContext(...) but I was wondering if there's a way to do this,
>> > e.g., attaching an appctx/dict to the pc itself. Does anyone have ideas
>> or
>> > suggestions on how to go about doing this?
>> >
>> > Thanks,
>> > Justin
>>

From maahi.buet at gmail.com  Tue Jan 22 21:11:19 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Tue, 22 Jan 2019 22:11:19 -0500
Subject: [petsc-users] Accessing the elements of a Matrix
In-Reply-To: <20BBAEAE-B44E-4FDC-977B-9A892857226D@anl.gov>
References: <CAJOaupHtT1yJEYQ=O+01mf2+F82y7=QWc02WZkJf=5AGNDY6fQ@mail.gmail.com>
	<20BBAEAE-B44E-4FDC-977B-9A892857226D@anl.gov>
Message-ID: <CAJOaupEz5jFbAJoeEsqSeSwG49YuxXc2Dw1zfb+EjO-nt-Y2oA@mail.gmail.com>

Thank you all !


On Tue, Jan 22, 2019 at 4:30 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    MatGetArrayF90 is an out-dated routine. For dense matrices you can use
> MatDenseGetArrayF90()
>
>   For sparse matrices you can use MatGetValues() or MatGetRow()
>
>    Barry
>
>
> > On Jan 22, 2019, at 3:15 PM, Maahi Talukder via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello All,
> >
> > I was trying to use ' MatGetArrayF90' to access a matrix and check its
> values. But I couldn't do it, probably because of the specific compiler I
> am using.
> > Is there any other way to access a matrix and check its values ?
> >
> >
> > Regards,
> > Maahi Talukder
> > Clarkson University
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190122/19f3c1ee/attachment.html>

From evanum at gmail.com  Wed Jan 23 02:24:36 2019
From: evanum at gmail.com (Evan Um)
Date: Wed, 23 Jan 2019 00:24:36 -0800
Subject: [petsc-users] Printing parallel matrix
Message-ID: <CAP1yThSYvjmZpvPZYN7Ef_P4+jp9o8VS6qDQj71foju2yv4oBA@mail.gmail.com>

Dear PETSC users,

I try to verify a matrix by printing a PETSC matrix and comparing its
elements with reference values. Below is my test code.

It works well when a single process is used. The output file is created
quickly. In contrast, when multiple processes (>2) are used, printing
matrix is stuck (the parallel matrix is assembled very quickly owing to
memory preallocation). The output file just prints two lines below. No
element is printed.

Mat Object: 2 MPI processes
  type: mpiaij

I assume that printing matrix is also designed for a parallel matrix. Does
this suggest that my parallel matrix includes any errors? Otherwise, does
this work only for a serial matrix?

Thanks for reading this question.

Regards,
Evan

MatCreateAIJ(PETSC_COMM_WORLD, m, n, M, N, 0, d_nnz_A, 0, o_nnz_A, &Mat_A);
MatSetFromOptions(Mat_A);

for (int i=rstart_A[rank]; i<rend_A[rank]; i++) {
mat_value=val_A[i]+PETSC_i*0.0;
MatSetValue(Mat_A, i_A[i],j_A[i],mat_value,ADD_VALUES);
}

MatAssemblyBegin(Mat_A, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(Mat_A, MAT_FINAL_ASSEMBLY);

PetscViewer viewer;
PetscViewerASCIIOpen(PETSC_COMM_WORLD,"Mat_A",&viewer);
MatView(Mat_A,viewer);
PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_MATLAB);
PetscViewerPopFormat(viewer);
PetscViewerDestroy(&viewer);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/f94b4405/attachment.html>

From tempohoper at gmail.com  Wed Jan 23 02:50:50 2019
From: tempohoper at gmail.com (Sal Am)
Date: Wed, 23 Jan 2019 08:50:50 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
	<CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>
Message-ID: <CAE3k7C4C4Ues6Ssu1SCPKYkPAdnMGE7n0itQDavbH1jKzTzCrA@mail.gmail.com>

Sorry it took long had to see if I could shrink down the problem files from
50GB to something smaller (now ~10GB).

> Can you compress your matrix and upload it to google drive, so we can try
> to reproduce the error.
>

How I ran the problem: mpiexec valgrind --tool=memcheck
--suppressions=$HOME/valgrind/valgrind-openmpi.supp -q --num-callers=20
--log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
-pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
-ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

here is the link to matrix A and vector b:
https://drive.google.com/drive/folders/16YQPTK6TfXC6pV5RMdJ9g7X-ZiqbvwU8?usp=sharing

I redid the problem (twice) by trying to solve a 1M finite elements problem
corresponding to ~ 4M n and 417M nnz matrix elements on the login shell
which has ~550GB mem, but it failed. First time it failed because of bus
error, second time it said killed. I have attached valgrind file from both
runs.

OpenMPI is not my favorite. You need to use a suppressions file to get rid
> of all of that noise. Here is one:
>

Thanks I have been using it, but sometimes I still see same amount of
errors.



On Fri, Jan 18, 2019 at 3:12 AM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

> Usually when I meet a SEGV error, I will run it again with a parallel
> debugger like DDT and wait for it to segfault, and then examine the stack
> trace to see what is wrong.
> Can you compress your matrix and upload it to google drive, so we can try
> to reproduce the error.
> --Junchao Zhang
>
>
> On Thu, Jan 17, 2019 at 10:44 AM Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> I did two runs, one with the SuperLU_dist and one with bcgs using jacobi,
>> attached are the results of one of the reports from valgrind on one random
>> processor (out of the 128 files).
>>
>> DS = direct solver
>> IS = iterative  solver
>>
>> There is an awful lot of errors.
>>
>> how I initiated the two runs:
>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>> --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs
>> -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce
>> -ksp_monitor -log_view
>>
>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
>> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>
>> Thank you
>>
>> On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:
>>>
>>>> 1) Running out of memory
>>>>>
>>>>> 2) You passed an invalid array
>>>>>
>>>> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e.
>>>> using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors
>>>> per node (128 processors in total).
>>>> I am not sure what would constitute an invalid array or how I can check
>>>> that. I am using the same procedure as when dealing with the smaller
>>>> matrix. i.e. Generate matrix A and vector b using FEM software then convert
>>>> the matrix and vector using a python script ready for petsc. read in petsc
>>>> and calculate.
>>>>
>>>> Are you running with 64-bit ints here?
>>>>>
>>>> Yes I have it configured petsc with  --with-64-bit-indices and
>>>> debugging mode, which this was run on.
>>>>
>>>
>>> It sounds like you have enough memory, but the fact that is runs for
>>> smaller problems makes me suspicious. It
>>> could still be a memory overwrite. Can you either
>>>
>>> a) Run under valgrind
>>>
>>> or
>>>
>>> b) Run under the debugger and get a stack trace
>>>
>>>  ?
>>>
>>>    Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>
>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>>
>>>>>> I will try that, although I think solving the real problem (later on
>>>>>> if I can get this to work) with 30 million finite elements might be a
>>>>>> problem for SuperLU_dist. so it is better to get an iterative solver to
>>>>>> work with first.
>>>>>>
>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>
>>>>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>>>>> -ksp_monitor -log_view
>>>>>> I have attached the full error output.
>>>>>>
>>>>>
>>>>> You are getting an SEGV on MatSetValues(), so its either
>>>>>
>>>>> 1) Running out of memory
>>>>>
>>>>> 2) You passed an invalid array
>>>>>
>>>>>
>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>>> None of our tests show this problem.
>>>>>>>
>>>>>> I am not how I can do that, but i have added my code which is quite
>>>>>> short and should only read and solve the system, the problem arises at
>>>>>> larger matrices for example current test case has 6 million finite elements
>>>>>> (~2B non-zero numbers and 25M x 25M matrix).
>>>>>>
>>>>>
>>>>> Are you running with 64-bit ints here?
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> The memory requested is an insane number. You may need to use 64
>>>>>>>>> bit integers.
>>>>>>>>
>>>>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the process
>>>>>>>> it says I can no longer use MUMPS and SuperLU as they are not supported (I
>>>>>>>> see on MUMPS webpage it supports 64int). However it does not exactly solve
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>
>>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>>
>>>>>>>
>>>>>>>> This time, it crashes at
>>>>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ()
>>>>>>>>> line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>>>>> which allocates local portion of B^T*A.
>>>>>>>>> You may also try to increase number of cores to reduce local
>>>>>>>>> matrix size.
>>>>>>>>>
>>>>>>>>
>>>>>>>> So I increased the number of cores to 16 on one node and ran it by :
>>>>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>>>>> It crashed after reading in the matrix and before starting to
>>>>>>>> solve. The error:
>>>>>>>>
>>>>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>>> the batch system) has told this process to end
>>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [2]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
>>>>>>>> the batch system) has told this process to end
>>>>>>>> [3]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [4]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [8]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [12]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01
>>>>>>>> exited on signal 9 (Killed).
>>>>>>>>
>>>>>>>> Now I was running this with valgrind as someone had previously
>>>>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>>>>
>>>>>>>
>>>>>>> Okay, its possible that there are bugs in the MPI implementation. So
>>>>>>>
>>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>
>>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>>> None of our tests show this problem.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> ==25940== Invalid read of size 8
>>>>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>> (mpits.c:373)
>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16
>>>>>>>> alloc'd
>>>>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>> (mpits.c:371)
>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>> ==25940==
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Fande:
>>>>>>>>>
>>>>>>>>>> According to this PR
>>>>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>>>>
>>>>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>>>>
>>>>>>>>> Sure, we can. But I feel we need do more tests to compare scalable
>>>>>>>>> and non-scalable algorithms.
>>>>>>>>> On theory, for small to medium matrices, non-scalable matmatmult()
>>>>>>>>> algorithm enables more efficient
>>>>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>>>>> Hong
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>>>>> Hong
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>>>>
>>>>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not properly
>>>>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>>>>
>>>>>>>>>>>> Probably the matrix is too large. You can try with more compute
>>>>>>>>>>>> nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>>>>
>>>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a
>>>>>>>>>>>>> matrix of 25M x 25M I get the following error:
>>>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below
>>>>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>>>>>>>>>>>> not available,
>>>>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start of
>>>>>>>>>>>>> the function
>>>>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line
>>>>>>>>>>>>> 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>>>>>>>>>>>> trouble shooting.
>>>>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named
>>>>>>>>>>>>> r02g03 by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>>>>> [4]PETSC ERROR: Configure options
>>>>>>>>>>>>> PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown
>>>>>>>>>>>>> file
>>>>>>>>>>>>>
>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
>>>>>>>>>>>>> with errorcode 59.
>>>>>>>>>>>>>
>>>>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>>>>> processes.
>>>>>>>>>>>>> You may or may not see output from other processes, depending
>>>>>>>>>>>>> on
>>>>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>
>>>>>>>>>>>>> Using Valgrind on only one of the valgrind files the following
>>>>>>>>>>>>> error was written:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays
>>>>>>>>>>>>> (aij.c:4445)
>>>>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>>>>> (recently) free'd
>>>>>>>>>>>>> ==9053==
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I ran
>>>>>>>>>>>>>> the code again with the following options:
>>>>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>>>>> gamg -log_view
>>>>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>>>>> the log_view.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error changes
>>>>>>>>>>>>>>>> slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This likely implies that you have a memory error in your
>>>>>>>>>>>>>>> code (a memory leak would not cause this behaviour).
>>>>>>>>>>>>>>> I strongly suggest you make sure your code is free of memory
>>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it
>>>>>>>>>>>>>>>> with -malloc_dump, but not getting anything printed out, however, when run
>>>>>>>>>>>>>>>> with -log_view I see that Viewer is created 4 times, but destroyed 3 times.
>>>>>>>>>>>>>>>> The way I see it, I have destroyed it where I see I no longer have use for
>>>>>>>>>>>>>>>> it so not sure if I am wrong. Could this be the reason why it keeps
>>>>>>>>>>>>>>>> crashing? It crashes as soon as it reads the matrix, before entering the
>>>>>>>>>>>>>>>> solving mode (I have a print statement before solving starts that never
>>>>>>>>>>>>>>>> prints).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> how I run it in the job script on 2 node with 32 processors
>>>>>>>>>>>>>>>> using the clusters OpenMPI.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/d6f37941/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind.log-DS.46544
Type: application/octet-stream
Size: 2905 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/d6f37941/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrind.log-DS-II.11295
Type: application/octet-stream
Size: 1393 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/d6f37941/attachment-0003.obj>

From jed at jedbrown.org  Wed Jan 23 09:40:32 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 23 Jan 2019 08:40:32 -0700
Subject: [petsc-users] Printing parallel matrix
In-Reply-To: <CAP1yThSYvjmZpvPZYN7Ef_P4+jp9o8VS6qDQj71foju2yv4oBA@mail.gmail.com>
References: <CAP1yThSYvjmZpvPZYN7Ef_P4+jp9o8VS6qDQj71foju2yv4oBA@mail.gmail.com>
Message-ID: <87r2d3idqn.fsf@jedbrown.org>

Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:

> Dear PETSC users,
>
> I try to verify a matrix by printing a PETSC matrix and comparing its
> elements with reference values. Below is my test code.
>
> It works well when a single process is used. The output file is created
> quickly. In contrast, when multiple processes (>2) are used, printing
> matrix is stuck (the parallel matrix is assembled very quickly owing to
> memory preallocation). The output file just prints two lines below. No
> element is printed.
>
> Mat Object: 2 MPI processes
>   type: mpiaij
>
> I assume that printing matrix is also designed for a parallel matrix. Does
> this suggest that my parallel matrix includes any errors? Otherwise, does
> this work only for a serial matrix?
>
> Thanks for reading this question.
>
> Regards,
> Evan
>
> MatCreateAIJ(PETSC_COMM_WORLD, m, n, M, N, 0, d_nnz_A, 0, o_nnz_A, &Mat_A);
> MatSetFromOptions(Mat_A);
>
> for (int i=rstart_A[rank]; i<rend_A[rank]; i++) {
> mat_value=val_A[i]+PETSC_i*0.0;
> MatSetValue(Mat_A, i_A[i],j_A[i],mat_value,ADD_VALUES);
> }
>
> MatAssemblyBegin(Mat_A, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd(Mat_A, MAT_FINAL_ASSEMBLY);
>
> PetscViewer viewer;
> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"Mat_A",&viewer);
> MatView(Mat_A,viewer);
> PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_MATLAB);

Did you intend to use the MATLAB format?  If so, MatView needs to be
called while the format is pushed, so swap the two lines above.

> PetscViewerPopFormat(viewer);
> PetscViewerDestroy(&viewer);

From chen2018 at purdue.edu  Wed Jan 23 12:27:12 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Wed, 23 Jan 2019 18:27:12 +0000
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
Message-ID: <DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>

Hi Matt,


I tried to modify the structure of present code and use KSP in the main program(optimal_mechanical_part).  Now the makefile is as following :

#================================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
HiDAC_Utils/HiDAC_Utils.f90 \
HiDAC_Material/HiDAC_baseMat.f90 \
HiDAC_Material/HiDAC_isoElasticMat.f90 \
HiDAC_Material/HiDAC_thermoMat.f90 \
HiDAC_Material/HiDAC_Material.f90 \
HiDAC_Approximation/HiDAC_node.f90 \
HiDAC_Approximation/HiDAC_nodeNet.f90 \
HiDAC_Approximation/HiDAC_baseApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Domain/HiDAC_enrichment.f90 \
HiDAC_Domain/HiDAC_composedDomain.f90 \
HiDAC_Domain/HiDAC_Domain.f90 \
HiDAC_Quadrature/HiDAC_quadType.f90 \
HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
HiDAC_Quadrature/HiDAC_Quadrature.f90 \
HiDAC_Analysis/HiDAC_BC.f90 \
HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
HiDAC_Analysis/HiDAC_elastostatics.f90 \
HiDAC_Analysis/HiDAC_heatConduction.f90 \
HiDAC_Analysis/HiDAC_phaseTransition.f90 \
HiDAC_Analysis/HiDAC_fracture.f90 \
HiDAC_Analysis/HiDAC_lineenrich.f90 \
HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
HiDAC_Analysis/HiDAC_Analysis.f90 \
HiDAC_Solver/HiDAC_baseSolver.f90 \
HiDAC_Solver/HiDAC_LapackSolver.f90 \
HiDAC_Solver/HiDAC_MumpsSolver.f90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $(OBJS_DIR)
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}

$(LIB): $(OBJS)
echo $(OBJS)
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)optimal_mechanical_part.o: \
optimal_mechanical_part.f90 \
$(OBJS_DIR)read_iges_file.o \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Domain.o \
$(OBJS_DIR)HiDAC_Material.o \
$(OBJS_DIR)HiDAC_Quadrature.o \
$(OBJS_DIR)HiDAC_Analysis.o \
$(OBJS_DIR)HiDAC_Utils.o

$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

$(OBJS_DIR)dxf_reader.o: \
dxf_reader.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)parse.o: \
parse.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)map.o: \
map.f90 \
$(OBJS_DIR)nrb.o

<Other dependencies >
# $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
# mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
# -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
# -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90 \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
#    This part is removed

endif

#============================Makefile finished ===============================



and the main program starts like this:


!============================  Main program starts=========================

program main
  !******************************************************!
  ! Import all HiDAC library
  !******************************************************!
  include 'mpif.h'
#include <petsc/finclude/petscksp.h>
  use petscksp
  use read_iges_file
  use HiDAC_Approximation
  use HiDAC_Domain
  use HiDAC_Material
  use HiDAC_Analysis
  use HiDAC_Utils
  use HiDAC_quadType

  implicit none
!=================================================================

I have not call the function in PETSC yet. Just add the KSP library in my main program. But I received  the following error when compiling:


HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


How can I solve this problem?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Friday, January 18, 2019 2:21 PM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Do not drop the Cc.


I have hundreds .F90 files to compile as well as an .c file to compile in this project. The present makefile is more complex than the PETSC makefile. Do you mean I should use the PETSC makefile as a template and add all the dependencies in it?

If you want to do it by hand you can, its just more error prone and verbose.


My present Makefile is like the following(Red parts are news added related to Petsc). Is this the right direction? Since a static pattern is defined while the compiling for PetscSolver is a little different. I guess I should list it as a specific case instead of using the pattern.

I will comment inline.


# ============================================================================
# Name        : Makefile
# Author      : Tao Song
# Version     : 2.1
# Copyright   : GNU 2.1
# Description : Makefile for HiDACX
# ============================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
......
<other .f90 files>
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Solver/HiDAC_PetscSolver.F90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

You do not need 'rules' if you are doing things by hand. Although it is easier to just use the builtin rule for building source.


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $OBJS_DIR
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)

$(LIB): $(OBJS)
echo $OBJS
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

<.... Other dependencies>

$ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
-I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
-I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90 \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
${PETSC_KSP_LIB}

You can do this, but to be certain you should should use

  ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}

along with your flags (-J, etc.). Notice that you do not need the *.o files here since you are only compiling.

You can simplify this by includeing 'rules' and just putting the $ (OBJS_DIR)HiDAC_PetscSolver.o as a
dependence for your link.

  Thanks,

    Matt

endif



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 11:57 AM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Now I have some problem with makefile.I want to integrate PETSC with a previous FEM Fortran code. To do that, I add a module named PetscSolver. It starts as the following:

It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the PETSc makefiles to avoid problems.

  Matt


!===================================================================

  module HiDAC_PetscSolver
#include <petsc/finclude/petscksp.h>
  use petscksp
  use HiDAC_Utils
  use HiDAC_baseSolver
    implicit none
  private
  type, extends(Solver), public  :: PetscSolver
    private
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
  contains
     procedure :: initialize => initialize_PetscSolver
     procedure :: assemble => assemble_PetscSolver
     procedure :: solve => solve_PetscSolver
     procedure :: postprocess => postprocess_PetscSolver
     procedure :: finalize => finalize_PetscSolver
     procedure :: getID
  end type PetscSolver
......
<Functions>
end module HiDAC_PetscSolver
!===================================================================
However, it did not compile successfully. It gave me the error:

 #include <petsc/finclude/petscksp.h>
  1
Warning: Illegal preprocessor directive
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:

      Vec              xvec,bvec,uvec
     1
Error: Unclassifiable statement at (1)
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:

      Mat              Amat
All function and type related to Petsc are not defined.

I must have made a very silly mistake. Can you show me how to fix it?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/c5cc0f1c/attachment-0001.html>

From knepley at gmail.com  Wed Jan 23 12:30:04 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 23 Jan 2019 13:30:04 -0500
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>

On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> Hi Matt,
>
>
> I tried to modify the structure of present code and use KSP in the main
> program(optimal_mechanical_part).  Now the makefile is as following :
>
>
> #================================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
> HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
> HiDAC_Utils/HiDAC_Utils.f90 \
> HiDAC_Material/HiDAC_baseMat.f90 \
> HiDAC_Material/HiDAC_isoElasticMat.f90 \
> HiDAC_Material/HiDAC_thermoMat.f90 \
> HiDAC_Material/HiDAC_Material.f90 \
> HiDAC_Approximation/HiDAC_node.f90 \
> HiDAC_Approximation/HiDAC_nodeNet.f90 \
> HiDAC_Approximation/HiDAC_baseApproximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Domain/HiDAC_enrichment.f90 \
> HiDAC_Domain/HiDAC_composedDomain.f90 \
> HiDAC_Domain/HiDAC_Domain.f90 \
> HiDAC_Quadrature/HiDAC_quadType.f90 \
> HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_Quadrature.f90 \
> HiDAC_Analysis/HiDAC_BC.f90 \
> HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
> HiDAC_Analysis/HiDAC_elastostatics.f90 \
> HiDAC_Analysis/HiDAC_heatConduction.f90 \
> HiDAC_Analysis/HiDAC_phaseTransition.f90 \
> HiDAC_Analysis/HiDAC_fracture.f90 \
> HiDAC_Analysis/HiDAC_lineenrich.f90 \
> HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
> HiDAC_Analysis/HiDAC_Analysis.f90 \
> HiDAC_Solver/HiDAC_baseSolver.f90 \
> HiDAC_Solver/HiDAC_LapackSolver.f90 \
> HiDAC_Solver/HiDAC_MumpsSolver.f90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $(OBJS_DIR)
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}
>
> $(LIB): $(OBJS)
> echo $(OBJS)
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)optimal_mechanical_part.o: \
> optimal_mechanical_part.f90 \
> $(OBJS_DIR)read_iges_file.o \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Domain.o \
> $(OBJS_DIR)HiDAC_Material.o \
> $(OBJS_DIR)HiDAC_Quadrature.o \
> $(OBJS_DIR)HiDAC_Analysis.o \
> $(OBJS_DIR)HiDAC_Utils.o
>
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> $(OBJS_DIR)dxf_reader.o: \
> dxf_reader.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)parse.o: \
> parse.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)map.o: \
> map.f90 \
> $(OBJS_DIR)nrb.o
>
> <Other dependencies >
> # $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> # mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> # -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> # -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> #    This part is removed
>
> endif
>
> #============================Makefile
> finished ===============================
>
>
>
> and the main program starts like this:
>
>
> !============================  Main program starts=========================
>
> program main
>   !******************************************************!
>   ! Import all HiDAC library
>   !******************************************************!
>   include 'mpif.h'
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use read_iges_file
>   use HiDAC_Approximation
>   use HiDAC_Domain
>   use HiDAC_Material
>   use HiDAC_Analysis
>   use HiDAC_Utils
>   use HiDAC_quadType
>
>   implicit none
> !====================*=============*================================
>
> I have not call the function in PETSC yet. Just add the KSP library in my
> main program. But I received  the following error when compiling:
>
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> How can I solve this problem?
>

You named the file

  optimal_mechanical_part.f90

but it has to be

  optimal_mechanical_part.F90

in order for the compiler to preprocess it.

   Matt


> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 2:21 PM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> Do not drop the Cc.
>
>
> I have hundreds .F90 files to compile as well as an .c file to compile in
> this project. The present makefile is more complex than the PETSC makefile.
> Do you mean I should use the PETSC makefile as a template and add all the
> dependencies in it?
>
>
> If you want to do it by hand you can, its just more error prone and
> verbose.
>
>
> My present Makefile is like the following(Red parts are news added related
> to Petsc). Is this the right direction? Since a static pattern is defined
> while the compiling for PetscSolver is a little different. I guess I should
> list it as a specific case instead of using the pattern.
>
>
> I will comment inline.
>
>
> #
> ============================================================================
> # Name        : Makefile
> # Author      : Tao Song
> # Version     : 2.1
> # Copyright   : GNU 2.1
> # Description : Makefile for HiDACX
> #
> ============================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> ......
> <other .f90 files>
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Solver/HiDAC_PetscSolver.F90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> You do not need 'rules' if you are doing things by hand. Although it is
> easier to just use the builtin rule for building source.
>
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $OBJS_DIR
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)
>
> $(LIB): $(OBJS)
> echo $OBJS
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> <.... Other dependencies>
>
> $ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> ${PETSC_KSP_LIB}
>
>
> You can do this, but to be certain you should should use
>
>   ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}
>
> along with your flags (-J, etc.). Notice that you do not need the *.o
> files here since you are only compiling.
>
> You can simplify this by includeing 'rules' and just putting the $
> (OBJS_DIR)HiDAC_PetscSolver.o as a
> dependence for your link.
>
>   Thanks,
>
>     Matt
>
>
> endif
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 11:57 AM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
> Now I have some problem with makefile.I want to integrate PETSC with a
> previous FEM Fortran code. To do that, I add a module named PetscSolver. It
> starts as the following:
>
>
> It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the
> PETSc makefiles to avoid problems.
>
>   Matt
>
>
> !===================================================================
>
>   module HiDAC_PetscSolver
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use HiDAC_Utils
>   use HiDAC_baseSolver
>     implicit none
>   private
>   type, extends(Solver), public  :: PetscSolver
>     private
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>   contains
>      procedure :: initialize => initialize_PetscSolver
>      procedure :: assemble => assemble_PetscSolver
>      procedure :: solve => solve_PetscSolver
>      procedure :: postprocess => postprocess_PetscSolver
>      procedure :: finalize => finalize_PetscSolver
>      procedure :: getID
>   end type PetscSolver
> ......
> <Functions>
> end module HiDAC_PetscSolver
> !===================================================================
> However, it did not compile successfully. It gave me the error:
>
>  #include <petsc/finclude/petscksp.h>
>   1
> Warning: Illegal preprocessor directive
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:
>
>       Vec              xvec,bvec,uvec
>      1
> Error: Unclassifiable statement at (1)
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:
>
>       Mat              Amat
> All function and type related to Petsc are not defined.
>
> I must have made a very silly mistake. Can you show me how to fix it?
>
> Thanks
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/1de68e92/attachment-0001.html>

From chen2018 at purdue.edu  Wed Jan 23 13:31:14 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Wed, 23 Jan 2019 19:31:14 +0000
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
Message-ID: <DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>

Hi Matt,

    I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90. But I still received the following error:

HiDACX/obj/debug/optimal_mechanical_part.o
mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90 -o HiDACX/obj/debug/optimal_mechanical_part.o
error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


Is there anything wrong with the flag?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Wednesday, January 23, 2019 1:30 PM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,


I tried to modify the structure of present code and use KSP in the main program(optimal_mechanical_part).  Now the makefile is as following :

#================================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
HiDAC_Utils/HiDAC_Utils.f90 \
HiDAC_Material/HiDAC_baseMat.f90 \
HiDAC_Material/HiDAC_isoElasticMat.f90 \
HiDAC_Material/HiDAC_thermoMat.f90 \
HiDAC_Material/HiDAC_Material.f90 \
HiDAC_Approximation/HiDAC_node.f90 \
HiDAC_Approximation/HiDAC_nodeNet.f90 \
HiDAC_Approximation/HiDAC_baseApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Domain/HiDAC_enrichment.f90 \
HiDAC_Domain/HiDAC_composedDomain.f90 \
HiDAC_Domain/HiDAC_Domain.f90 \
HiDAC_Quadrature/HiDAC_quadType.f90 \
HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
HiDAC_Quadrature/HiDAC_Quadrature.f90 \
HiDAC_Analysis/HiDAC_BC.f90 \
HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
HiDAC_Analysis/HiDAC_elastostatics.f90 \
HiDAC_Analysis/HiDAC_heatConduction.f90 \
HiDAC_Analysis/HiDAC_phaseTransition.f90 \
HiDAC_Analysis/HiDAC_fracture.f90 \
HiDAC_Analysis/HiDAC_lineenrich.f90 \
HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
HiDAC_Analysis/HiDAC_Analysis.f90 \
HiDAC_Solver/HiDAC_baseSolver.f90 \
HiDAC_Solver/HiDAC_LapackSolver.f90 \
HiDAC_Solver/HiDAC_MumpsSolver.f90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $(OBJS_DIR)
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}

$(LIB): $(OBJS)
echo $(OBJS)
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)optimal_mechanical_part.o: \
optimal_mechanical_part.f90 \
$(OBJS_DIR)read_iges_file.o \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Domain.o \
$(OBJS_DIR)HiDAC_Material.o \
$(OBJS_DIR)HiDAC_Quadrature.o \
$(OBJS_DIR)HiDAC_Analysis.o \
$(OBJS_DIR)HiDAC_Utils.o

$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

$(OBJS_DIR)dxf_reader.o: \
dxf_reader.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)parse.o: \
parse.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)map.o: \
map.f90 \
$(OBJS_DIR)nrb.o

<Other dependencies >
# $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
# mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
# -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
# -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90 \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
#    This part is removed

endif

#============================Makefile finished ===============================



and the main program starts like this:


!============================  Main program starts=========================

program main
  !******************************************************!
  ! Import all HiDAC library
  !******************************************************!
  include 'mpif.h'
#include <petsc/finclude/petscksp.h>
  use petscksp
  use read_iges_file
  use HiDAC_Approximation
  use HiDAC_Domain
  use HiDAC_Material
  use HiDAC_Analysis
  use HiDAC_Utils
  use HiDAC_quadType

  implicit none
!=================================================================

I have not call the function in PETSC yet. Just add the KSP library in my main program. But I received  the following error when compiling:


HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


How can I solve this problem?

You named the file

  optimal_mechanical_part.f90

but it has to be

  optimal_mechanical_part.F90

in order for the compiler to preprocess it.

   Matt


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 2:21 PM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Do not drop the Cc.


I have hundreds .F90 files to compile as well as an .c file to compile in this project. The present makefile is more complex than the PETSC makefile. Do you mean I should use the PETSC makefile as a template and add all the dependencies in it?

If you want to do it by hand you can, its just more error prone and verbose.


My present Makefile is like the following(Red parts are news added related to Petsc). Is this the right direction? Since a static pattern is defined while the compiling for PetscSolver is a little different. I guess I should list it as a specific case instead of using the pattern.

I will comment inline.


# ============================================================================
# Name        : Makefile
# Author      : Tao Song
# Version     : 2.1
# Copyright   : GNU 2.1
# Description : Makefile for HiDACX
# ============================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
......
<other .f90 files>
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Solver/HiDAC_PetscSolver.F90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

You do not need 'rules' if you are doing things by hand. Although it is easier to just use the builtin rule for building source.


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $OBJS_DIR
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)

$(LIB): $(OBJS)
echo $OBJS
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

<.... Other dependencies>

$ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
-I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
-I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90 \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
${PETSC_KSP_LIB}

You can do this, but to be certain you should should use

  ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}

along with your flags (-J, etc.). Notice that you do not need the *.o files here since you are only compiling.

You can simplify this by includeing 'rules' and just putting the $ (OBJS_DIR)HiDAC_PetscSolver.o as a
dependence for your link.

  Thanks,

    Matt

endif



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 11:57 AM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Now I have some problem with makefile.I want to integrate PETSC with a previous FEM Fortran code. To do that, I add a module named PetscSolver. It starts as the following:

It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the PETSc makefiles to avoid problems.

  Matt


!===================================================================

  module HiDAC_PetscSolver
#include <petsc/finclude/petscksp.h>
  use petscksp
  use HiDAC_Utils
  use HiDAC_baseSolver
    implicit none
  private
  type, extends(Solver), public  :: PetscSolver
    private
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
  contains
     procedure :: initialize => initialize_PetscSolver
     procedure :: assemble => assemble_PetscSolver
     procedure :: solve => solve_PetscSolver
     procedure :: postprocess => postprocess_PetscSolver
     procedure :: finalize => finalize_PetscSolver
     procedure :: getID
  end type PetscSolver
......
<Functions>
end module HiDAC_PetscSolver
!===================================================================
However, it did not compile successfully. It gave me the error:

 #include <petsc/finclude/petscksp.h>
  1
Warning: Illegal preprocessor directive
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:

      Vec              xvec,bvec,uvec
     1
Error: Unclassifiable statement at (1)
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:

      Mat              Amat
All function and type related to Petsc are not defined.

I must have made a very silly mistake. Can you show me how to fix it?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190123/df6e28b9/attachment-0001.html>

From arkhazali at cc.iut.ac.ir  Thu Jan 24 03:53:12 2019
From: arkhazali at cc.iut.ac.ir (Ali Reza Khaz'ali)
Date: Thu, 24 Jan 2019 13:23:12 +0330
Subject: [petsc-users] Allocating for existing space on Mat
Message-ID: <000001d4b3ca$9ee3ec50$dcabc4f0$@cc.iut.ac.ir>

Hi all,

 

As far as I understand, using MatSetValues for new non-zero entries will
cause an error if MAT_NEW_NONZERO_LOCATION_ERR== PETSC_TRUE.  Setting
MAT_NEW_NONZERO_LOCATION_ERR to PETSC_FALSE is also undesirable since it
will slow down the code, and in some cases, the performance loss will be
extreme. However, is there any solution for a matrix that the number of
non-zeros on each of its rows is fixed in both diagonal and off-diagonal
parts, but the location (the column number) of the non-zero entries may vary
during the procedures? I can use MAT_NEW_NONZERO_LOCATION_ERR== PETSC_TRUE,
but it is important to me that the code runs as fast as possible.

Any help is much appreciated.

 

Best wishes,

Ali

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190124/9b32a0d8/attachment.html>

From knepley at gmail.com  Thu Jan 24 04:54:52 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Jan 2019 05:54:52 -0500
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
	<DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4Gko_pmpYfBi9qewWmrS_je3xdcygvDny7zhUJxcA0hnRg@mail.gmail.com>

On Wed, Jan 23, 2019 at 2:31 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> Hi Matt,
>
>     I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90.
> But I still received the following error:
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
> error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> Is there anything wrong with the flag?
>

You are compiling a Fortran file with mpicc, instead of mpif90?

   Matt


> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Wednesday, January 23, 2019 1:30 PM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> I tried to modify the structure of present code and use KSP in the main
> program(optimal_mechanical_part).  Now the makefile is as following :
>
>
> #================================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
> HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
> HiDAC_Utils/HiDAC_Utils.f90 \
> HiDAC_Material/HiDAC_baseMat.f90 \
> HiDAC_Material/HiDAC_isoElasticMat.f90 \
> HiDAC_Material/HiDAC_thermoMat.f90 \
> HiDAC_Material/HiDAC_Material.f90 \
> HiDAC_Approximation/HiDAC_node.f90 \
> HiDAC_Approximation/HiDAC_nodeNet.f90 \
> HiDAC_Approximation/HiDAC_baseApproximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Domain/HiDAC_enrichment.f90 \
> HiDAC_Domain/HiDAC_composedDomain.f90 \
> HiDAC_Domain/HiDAC_Domain.f90 \
> HiDAC_Quadrature/HiDAC_quadType.f90 \
> HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_Quadrature.f90 \
> HiDAC_Analysis/HiDAC_BC.f90 \
> HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
> HiDAC_Analysis/HiDAC_elastostatics.f90 \
> HiDAC_Analysis/HiDAC_heatConduction.f90 \
> HiDAC_Analysis/HiDAC_phaseTransition.f90 \
> HiDAC_Analysis/HiDAC_fracture.f90 \
> HiDAC_Analysis/HiDAC_lineenrich.f90 \
> HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
> HiDAC_Analysis/HiDAC_Analysis.f90 \
> HiDAC_Solver/HiDAC_baseSolver.f90 \
> HiDAC_Solver/HiDAC_LapackSolver.f90 \
> HiDAC_Solver/HiDAC_MumpsSolver.f90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $(OBJS_DIR)
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}
>
> $(LIB): $(OBJS)
> echo $(OBJS)
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)optimal_mechanical_part.o: \
> optimal_mechanical_part.f90 \
> $(OBJS_DIR)read_iges_file.o \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Domain.o \
> $(OBJS_DIR)HiDAC_Material.o \
> $(OBJS_DIR)HiDAC_Quadrature.o \
> $(OBJS_DIR)HiDAC_Analysis.o \
> $(OBJS_DIR)HiDAC_Utils.o
>
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> $(OBJS_DIR)dxf_reader.o: \
> dxf_reader.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)parse.o: \
> parse.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)map.o: \
> map.f90 \
> $(OBJS_DIR)nrb.o
>
> <Other dependencies >
> # $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> # mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> # -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> # -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> #    This part is removed
>
> endif
>
> #============================Makefile
> finished ===============================
>
>
>
> and the main program starts like this:
>
>
> !============================  Main program starts=========================
>
> program main
>   !******************************************************!
>   ! Import all HiDAC library
>   !******************************************************!
>   include 'mpif.h'
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use read_iges_file
>   use HiDAC_Approximation
>   use HiDAC_Domain
>   use HiDAC_Material
>   use HiDAC_Analysis
>   use HiDAC_Utils
>   use HiDAC_quadType
>
>   implicit none
> !====================*=============*================================
>
> I have not call the function in PETSC yet. Just add the KSP library in my
> main program. But I received  the following error when compiling:
>
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> How can I solve this problem?
>
>
> You named the file
>
>   optimal_mechanical_part.f90
>
> but it has to be
>
>   optimal_mechanical_part.F90
>
> in order for the compiler to preprocess it.
>
>    Matt
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 2:21 PM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> Do not drop the Cc.
>
>
> I have hundreds .F90 files to compile as well as an .c file to compile in
> this project. The present makefile is more complex than the PETSC makefile.
> Do you mean I should use the PETSC makefile as a template and add all the
> dependencies in it?
>
>
> If you want to do it by hand you can, its just more error prone and
> verbose.
>
>
> My present Makefile is like the following(Red parts are news added related
> to Petsc). Is this the right direction? Since a static pattern is defined
> while the compiling for PetscSolver is a little different. I guess I should
> list it as a specific case instead of using the pattern.
>
>
> I will comment inline.
>
>
> #
> ============================================================================
> # Name        : Makefile
> # Author      : Tao Song
> # Version     : 2.1
> # Copyright   : GNU 2.1
> # Description : Makefile for HiDACX
> #
> ============================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> ......
> <other .f90 files>
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Solver/HiDAC_PetscSolver.F90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> You do not need 'rules' if you are doing things by hand. Although it is
> easier to just use the builtin rule for building source.
>
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $OBJS_DIR
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)
>
> $(LIB): $(OBJS)
> echo $OBJS
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> <.... Other dependencies>
>
> $ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> ${PETSC_KSP_LIB}
>
>
> You can do this, but to be certain you should should use
>
>   ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}
>
> along with your flags (-J, etc.). Notice that you do not need the *.o
> files here since you are only compiling.
>
> You can simplify this by includeing 'rules' and just putting the $
> (OBJS_DIR)HiDAC_PetscSolver.o as a
> dependence for your link.
>
>   Thanks,
>
>     Matt
>
>
> endif
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 11:57 AM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
> Now I have some problem with makefile.I want to integrate PETSC with a
> previous FEM Fortran code. To do that, I add a module named PetscSolver. It
> starts as the following:
>
>
> It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the
> PETSc makefiles to avoid problems.
>
>   Matt
>
>
> !===================================================================
>
>   module HiDAC_PetscSolver
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use HiDAC_Utils
>   use HiDAC_baseSolver
>     implicit none
>   private
>   type, extends(Solver), public  :: PetscSolver
>     private
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>   contains
>      procedure :: initialize => initialize_PetscSolver
>      procedure :: assemble => assemble_PetscSolver
>      procedure :: solve => solve_PetscSolver
>      procedure :: postprocess => postprocess_PetscSolver
>      procedure :: finalize => finalize_PetscSolver
>      procedure :: getID
>   end type PetscSolver
> ......
> <Functions>
> end module HiDAC_PetscSolver
> !===================================================================
> However, it did not compile successfully. It gave me the error:
>
>  #include <petsc/finclude/petscksp.h>
>   1
> Warning: Illegal preprocessor directive
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:
>
>       Vec              xvec,bvec,uvec
>      1
> Error: Unclassifiable statement at (1)
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:
>
>       Mat              Amat
> All function and type related to Petsc are not defined.
>
> I must have made a very silly mistake. Can you show me how to fix it?
>
> Thanks
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190124/da9f6480/attachment-0001.html>

From knepley at gmail.com  Thu Jan 24 05:23:31 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Jan 2019 06:23:31 -0500
Subject: [petsc-users] Allocating for existing space on Mat
In-Reply-To: <000001d4b3ca$9ee3ec50$dcabc4f0$@cc.iut.ac.ir>
References: <000001d4b3ca$9ee3ec50$dcabc4f0$@cc.iut.ac.ir>
Message-ID: <CAMYG4G=N=wxz+7+uwJFAF=j-Jg=ctyy0w34xJW-+_xKvWrEp6w@mail.gmail.com>

On Thu, Jan 24, 2019 at 4:53 AM Ali Reza Khaz'ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi all,
>
>
>
> As far as I understand, using MatSetValues for new non-zero entries will
> cause an error if MAT_NEW_NONZERO_LOCATION_ERR== PETSC_TRUE.  Setting
> MAT_NEW_NONZERO_LOCATION_ERR to PETSC_FALSE is also undesirable since it
> will slow down the code, and in some cases, the performance loss will be
> extreme. However, is there any solution for a matrix that the number of
> non-zeros on each of its rows is fixed in both diagonal and off-diagonal
> parts, but the location (the column number) of the non-zero entries may
> vary during the procedures? I can use MAT_NEW_NONZERO_LOCATION_ERR==
> PETSC_TRUE, but it is important to me that the code runs as fast as
> possible.
>
> Any help is much appreciated.
>

The main reason for slowdown is reallocation. However, MatAssembly will
squeeze out extra space in all rows.
Thus, in order to move entries around without reallocating, you must stick
zeros there when they are no used.
I would do this:

  1) Measure the performance degradation from doing this naively

  2) If it is noticeable, then go in and write a MatChangeRow() that allows
you to alter the column index and value.

  Thanks,

     Matt


>
>
> Best wishes,
>
> Ali
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190124/e112b5cb/attachment.html>

From chen2018 at purdue.edu  Thu Jan 24 09:11:24 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Thu, 24 Jan 2019 15:11:24 +0000
Subject: [petsc-users] Fw: Integrate PETSC with existing Fortran
In-Reply-To: <DM5PR22MB02657A04F2A8BFFCA246E494A89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
	<DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CAMYG4Gko_pmpYfBi9qewWmrS_je3xdcygvDny7zhUJxcA0hnRg@mail.gmail.com>,
	<DM5PR22MB02657A04F2A8BFFCA246E494A89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <DM5PR22MB02650936E5F7715A61472E7CA89A0@DM5PR22MB0265.namprd22.prod.outlook.com>

However, I changed the makefile,but I still received the following error:


HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.

________________________________
From: Yaxiong Chen
Sent: Thursday, January 24, 2019 9:50 AM
To: Matthew Knepley
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran


I see. This is because I have one .c file in the bunch of files.

!==============================================================

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
This command in makefile will use CC  instead of FC since I change f90 to F90.


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, January 24, 2019 5:54 AM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Wed, Jan 23, 2019 at 2:31 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

    I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90. But I still received the following error:

HiDACX/obj/debug/optimal_mechanical_part.o
mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90 -o HiDACX/obj/debug/optimal_mechanical_part.o
error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


Is there anything wrong with the flag?

You are compiling a Fortran file with mpicc, instead of mpif90?

   Matt


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, January 23, 2019 1:30 PM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,


I tried to modify the structure of present code and use KSP in the main program(optimal_mechanical_part).  Now the makefile is as following :

#================================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
HiDAC_Utils/HiDAC_Utils.f90 \
HiDAC_Material/HiDAC_baseMat.f90 \
HiDAC_Material/HiDAC_isoElasticMat.f90 \
HiDAC_Material/HiDAC_thermoMat.f90 \
HiDAC_Material/HiDAC_Material.f90 \
HiDAC_Approximation/HiDAC_node.f90 \
HiDAC_Approximation/HiDAC_nodeNet.f90 \
HiDAC_Approximation/HiDAC_baseApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Domain/HiDAC_enrichment.f90 \
HiDAC_Domain/HiDAC_composedDomain.f90 \
HiDAC_Domain/HiDAC_Domain.f90 \
HiDAC_Quadrature/HiDAC_quadType.f90 \
HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
HiDAC_Quadrature/HiDAC_Quadrature.f90 \
HiDAC_Analysis/HiDAC_BC.f90 \
HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
HiDAC_Analysis/HiDAC_elastostatics.f90 \
HiDAC_Analysis/HiDAC_heatConduction.f90 \
HiDAC_Analysis/HiDAC_phaseTransition.f90 \
HiDAC_Analysis/HiDAC_fracture.f90 \
HiDAC_Analysis/HiDAC_lineenrich.f90 \
HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
HiDAC_Analysis/HiDAC_Analysis.f90 \
HiDAC_Solver/HiDAC_baseSolver.f90 \
HiDAC_Solver/HiDAC_LapackSolver.f90 \
HiDAC_Solver/HiDAC_MumpsSolver.f90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $(OBJS_DIR)
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}

$(LIB): $(OBJS)
echo $(OBJS)
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)optimal_mechanical_part.o: \
optimal_mechanical_part.f90 \
$(OBJS_DIR)read_iges_file.o \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Domain.o \
$(OBJS_DIR)HiDAC_Material.o \
$(OBJS_DIR)HiDAC_Quadrature.o \
$(OBJS_DIR)HiDAC_Analysis.o \
$(OBJS_DIR)HiDAC_Utils.o

$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

$(OBJS_DIR)dxf_reader.o: \
dxf_reader.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)parse.o: \
parse.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)map.o: \
map.f90 \
$(OBJS_DIR)nrb.o

<Other dependencies >
# $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
# mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
# -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
# -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90 \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
#    This part is removed

endif

#============================Makefile finished ===============================



and the main program starts like this:


!============================  Main program starts=========================

program main
  !******************************************************!
  ! Import all HiDAC library
  !******************************************************!
  include 'mpif.h'
#include <petsc/finclude/petscksp.h>
  use petscksp
  use read_iges_file
  use HiDAC_Approximation
  use HiDAC_Domain
  use HiDAC_Material
  use HiDAC_Analysis
  use HiDAC_Utils
  use HiDAC_quadType

  implicit none
!=================================================================

I have not call the function in PETSC yet. Just add the KSP library in my main program. But I received  the following error when compiling:


HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


How can I solve this problem?

You named the file

  optimal_mechanical_part.f90

but it has to be

  optimal_mechanical_part.F90

in order for the compiler to preprocess it.

   Matt


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 2:21 PM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Do not drop the Cc.


I have hundreds .F90 files to compile as well as an .c file to compile in this project. The present makefile is more complex than the PETSC makefile. Do you mean I should use the PETSC makefile as a template and add all the dependencies in it?

If you want to do it by hand you can, its just more error prone and verbose.


My present Makefile is like the following(Red parts are news added related to Petsc). Is this the right direction? Since a static pattern is defined while the compiling for PetscSolver is a little different. I guess I should list it as a specific case instead of using the pattern.

I will comment inline.


# ============================================================================
# Name        : Makefile
# Author      : Tao Song
# Version     : 2.1
# Copyright   : GNU 2.1
# Description : Makefile for HiDACX
# ============================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
......
<other .f90 files>
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Solver/HiDAC_PetscSolver.F90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

You do not need 'rules' if you are doing things by hand. Although it is easier to just use the builtin rule for building source.


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $OBJS_DIR
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)

$(LIB): $(OBJS)
echo $OBJS
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

<.... Other dependencies>

$ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
-I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
-I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90 \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
${PETSC_KSP_LIB}

You can do this, but to be certain you should should use

  ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}

along with your flags (-J, etc.). Notice that you do not need the *.o files here since you are only compiling.

You can simplify this by includeing 'rules' and just putting the $ (OBJS_DIR)HiDAC_PetscSolver.o as a
dependence for your link.

  Thanks,

    Matt

endif



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 11:57 AM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Now I have some problem with makefile.I want to integrate PETSC with a previous FEM Fortran code. To do that, I add a module named PetscSolver. It starts as the following:

It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the PETSc makefiles to avoid problems.

  Matt


!===================================================================

  module HiDAC_PetscSolver
#include <petsc/finclude/petscksp.h>
  use petscksp
  use HiDAC_Utils
  use HiDAC_baseSolver
    implicit none
  private
  type, extends(Solver), public  :: PetscSolver
    private
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
  contains
     procedure :: initialize => initialize_PetscSolver
     procedure :: assemble => assemble_PetscSolver
     procedure :: solve => solve_PetscSolver
     procedure :: postprocess => postprocess_PetscSolver
     procedure :: finalize => finalize_PetscSolver
     procedure :: getID
  end type PetscSolver
......
<Functions>
end module HiDAC_PetscSolver
!===================================================================
However, it did not compile successfully. It gave me the error:

 #include <petsc/finclude/petscksp.h>
  1
Warning: Illegal preprocessor directive
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:

      Vec              xvec,bvec,uvec
     1
Error: Unclassifiable statement at (1)
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:

      Mat              Amat
All function and type related to Petsc are not defined.

I must have made a very silly mistake. Can you show me how to fix it?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190124/2db00b15/attachment-0001.html>

From knepley at gmail.com  Thu Jan 24 09:43:02 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 24 Jan 2019 10:43:02 -0500
Subject: [petsc-users] Fw: Integrate PETSC with existing Fortran
In-Reply-To: <DM5PR22MB02650936E5F7715A61472E7CA89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
	<DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gko_pmpYfBi9qewWmrS_je3xdcygvDny7zhUJxcA0hnRg@mail.gmail.com>
	<DM5PR22MB02657A04F2A8BFFCA246E494A89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<DM5PR22MB02650936E5F7715A61472E7CA89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4GmaP=NvNOo0iuRQsxQkF1Eoqcb3tkkWTwMK6z-SRdqJEg@mail.gmail.com>

On Thu, Jan 24, 2019 at 10:11 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> However, I changed the makefile,but I still received the following error:
>

Your includes are still not right:

master $:/PETSc3/petsc/petsc-dev$ make getincludedirs
make getincludedirs
-I/PETSc3/petsc/petsc-dev/include
-I/PETSc3/petsc/petsc-dev/arch-master-debug/include
-I/PETSc3/petsc/petsc-dev/arch-master-debug/include/eigen3
-I/opt/X11/include -I/PETSc3/petsc/include

You are missing the one for the $PETSC_ARCH include directory.

   Matt


> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
>
> ------------------------------
> *From:* Yaxiong Chen
> *Sent:* Thursday, January 24, 2019 9:50 AM
> *To:* Matthew Knepley
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
>
> I see. This is because I have one .c file in the bunch of files.
>
> !==============================================================
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
> This command in makefile will use CC  instead of FC since I change f90 to
> F90.
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, January 24, 2019 5:54 AM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Wed, Jan 23, 2019 at 2:31 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>     I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90.
> But I still received the following error:
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
> error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> Is there anything wrong with the flag?
>
>
> You are compiling a Fortran file with mpicc, instead of mpif90?
>
>    Matt
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Wednesday, January 23, 2019 1:30 PM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> I tried to modify the structure of present code and use KSP in the main
> program(optimal_mechanical_part).  Now the makefile is as following :
>
>
> #================================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
> HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
> HiDAC_Utils/HiDAC_Utils.f90 \
> HiDAC_Material/HiDAC_baseMat.f90 \
> HiDAC_Material/HiDAC_isoElasticMat.f90 \
> HiDAC_Material/HiDAC_thermoMat.f90 \
> HiDAC_Material/HiDAC_Material.f90 \
> HiDAC_Approximation/HiDAC_node.f90 \
> HiDAC_Approximation/HiDAC_nodeNet.f90 \
> HiDAC_Approximation/HiDAC_baseApproximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Domain/HiDAC_enrichment.f90 \
> HiDAC_Domain/HiDAC_composedDomain.f90 \
> HiDAC_Domain/HiDAC_Domain.f90 \
> HiDAC_Quadrature/HiDAC_quadType.f90 \
> HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_Quadrature.f90 \
> HiDAC_Analysis/HiDAC_BC.f90 \
> HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
> HiDAC_Analysis/HiDAC_elastostatics.f90 \
> HiDAC_Analysis/HiDAC_heatConduction.f90 \
> HiDAC_Analysis/HiDAC_phaseTransition.f90 \
> HiDAC_Analysis/HiDAC_fracture.f90 \
> HiDAC_Analysis/HiDAC_lineenrich.f90 \
> HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
> HiDAC_Analysis/HiDAC_Analysis.f90 \
> HiDAC_Solver/HiDAC_baseSolver.f90 \
> HiDAC_Solver/HiDAC_LapackSolver.f90 \
> HiDAC_Solver/HiDAC_MumpsSolver.f90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $(OBJS_DIR)
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}
>
> $(LIB): $(OBJS)
> echo $(OBJS)
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)optimal_mechanical_part.o: \
> optimal_mechanical_part.f90 \
> $(OBJS_DIR)read_iges_file.o \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Domain.o \
> $(OBJS_DIR)HiDAC_Material.o \
> $(OBJS_DIR)HiDAC_Quadrature.o \
> $(OBJS_DIR)HiDAC_Analysis.o \
> $(OBJS_DIR)HiDAC_Utils.o
>
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> $(OBJS_DIR)dxf_reader.o: \
> dxf_reader.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)parse.o: \
> parse.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)map.o: \
> map.f90 \
> $(OBJS_DIR)nrb.o
>
> <Other dependencies >
> # $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> # mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> # -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> # -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> #    This part is removed
>
> endif
>
> #============================Makefile
> finished ===============================
>
>
>
> and the main program starts like this:
>
>
> !============================  Main program starts=========================
>
> program main
>   !******************************************************!
>   ! Import all HiDAC library
>   !******************************************************!
>   include 'mpif.h'
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use read_iges_file
>   use HiDAC_Approximation
>   use HiDAC_Domain
>   use HiDAC_Material
>   use HiDAC_Analysis
>   use HiDAC_Utils
>   use HiDAC_quadType
>
>   implicit none
> !====================*=============*================================
>
> I have not call the function in PETSC yet. Just add the KSP library in my
> main program. But I received  the following error when compiling:
>
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> How can I solve this problem?
>
>
> You named the file
>
>   optimal_mechanical_part.f90
>
> but it has to be
>
>   optimal_mechanical_part.F90
>
> in order for the compiler to preprocess it.
>
>    Matt
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 2:21 PM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> Do not drop the Cc.
>
>
> I have hundreds .F90 files to compile as well as an .c file to compile in
> this project. The present makefile is more complex than the PETSC makefile.
> Do you mean I should use the PETSC makefile as a template and add all the
> dependencies in it?
>
>
> If you want to do it by hand you can, its just more error prone and
> verbose.
>
>
> My present Makefile is like the following(Red parts are news added related
> to Petsc). Is this the right direction? Since a static pattern is defined
> while the compiling for PetscSolver is a little different. I guess I should
> list it as a specific case instead of using the pattern.
>
>
> I will comment inline.
>
>
> #
> ============================================================================
> # Name        : Makefile
> # Author      : Tao Song
> # Version     : 2.1
> # Copyright   : GNU 2.1
> # Description : Makefile for HiDACX
> #
> ============================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> ......
> <other .f90 files>
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Solver/HiDAC_PetscSolver.F90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> You do not need 'rules' if you are doing things by hand. Although it is
> easier to just use the builtin rule for building source.
>
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $OBJS_DIR
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)
>
> $(LIB): $(OBJS)
> echo $OBJS
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> <.... Other dependencies>
>
> $ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> ${PETSC_KSP_LIB}
>
>
> You can do this, but to be certain you should should use
>
>   ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}
>
> along with your flags (-J, etc.). Notice that you do not need the *.o
> files here since you are only compiling.
>
> You can simplify this by includeing 'rules' and just putting the $
> (OBJS_DIR)HiDAC_PetscSolver.o as a
> dependence for your link.
>
>   Thanks,
>
>     Matt
>
>
> endif
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 11:57 AM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
> Now I have some problem with makefile.I want to integrate PETSC with a
> previous FEM Fortran code. To do that, I add a module named PetscSolver. It
> starts as the following:
>
>
> It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the
> PETSc makefiles to avoid problems.
>
>   Matt
>
>
> !===================================================================
>
>   module HiDAC_PetscSolver
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use HiDAC_Utils
>   use HiDAC_baseSolver
>     implicit none
>   private
>   type, extends(Solver), public  :: PetscSolver
>     private
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>   contains
>      procedure :: initialize => initialize_PetscSolver
>      procedure :: assemble => assemble_PetscSolver
>      procedure :: solve => solve_PetscSolver
>      procedure :: postprocess => postprocess_PetscSolver
>      procedure :: finalize => finalize_PetscSolver
>      procedure :: getID
>   end type PetscSolver
> ......
> <Functions>
> end module HiDAC_PetscSolver
> !===================================================================
> However, it did not compile successfully. It gave me the error:
>
>  #include <petsc/finclude/petscksp.h>
>   1
> Warning: Illegal preprocessor directive
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:
>
>       Vec              xvec,bvec,uvec
>      1
> Error: Unclassifiable statement at (1)
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:
>
>       Mat              Amat
> All function and type related to Petsc are not defined.
>
> I must have made a very silly mistake. Can you show me how to fix it?
>
> Thanks
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190124/e56974de/attachment-0001.html>

From hsuh7 at gatech.edu  Thu Jan 24 14:57:44 2019
From: hsuh7 at gatech.edu (David)
Date: Thu, 24 Jan 2019 20:57:44 +0000
Subject: [petsc-users] [tao] General L1,L2 optimization
Message-ID: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>

Hi. I was wondering whether there was some kind of general consensus about

the currently-best-implemented L1, L2 norm regularization for petsc/tao 
that has been implemented.

Naively, I would shoot for Levenberg-Marquardt for some kind of random 
matrix, or even generic

finite-difference stencil problem. (but it seems like LM is yet to be 
implemented, but only on petsc manual pdf?)

Or perhaps, of the implemented ones, LMVM seems to work well, at least 
on my local machine.

In any due case, I would highly appreciate the input and opinion about 
these matters.


Thanks.



Hansol Suh,

PhD Student


Georgia Institute of Technology


From bsmith at mcs.anl.gov  Thu Jan 24 21:58:08 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 25 Jan 2019 03:58:08 +0000
Subject: [petsc-users] Printing parallel matrix
In-Reply-To: <87r2d3idqn.fsf@jedbrown.org>
References: <CAP1yThSYvjmZpvPZYN7Ef_P4+jp9o8VS6qDQj71foju2yv4oBA@mail.gmail.com>
	<87r2d3idqn.fsf@jedbrown.org>
Message-ID: <F9FF5D27-C487-4BD4-AFF9-424D088E40EE@anl.gov>


  Is it a very large matrix? The ASCII output is intended for only smallish matrices and may take a lot of time for very large matrices. For large matrices we recommend the binary format which is much faster for large matrices and can handle tens of millions of rows. You can read it into Python or Matlab with utilities we provide.

    Barry


> On Jan 23, 2019, at 9:40 AM, Jed Brown via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:
> 
>> Dear PETSC users,
>> 
>> I try to verify a matrix by printing a PETSC matrix and comparing its
>> elements with reference values. Below is my test code.
>> 
>> It works well when a single process is used. The output file is created
>> quickly. In contrast, when multiple processes (>2) are used, printing
>> matrix is stuck (the parallel matrix is assembled very quickly owing to
>> memory preallocation). The output file just prints two lines below. No
>> element is printed.
>> 
>> Mat Object: 2 MPI processes
>>  type: mpiaij
>> 
>> I assume that printing matrix is also designed for a parallel matrix. Does
>> this suggest that my parallel matrix includes any errors? Otherwise, does
>> this work only for a serial matrix?
>> 
>> Thanks for reading this question.
>> 
>> Regards,
>> Evan
>> 
>> MatCreateAIJ(PETSC_COMM_WORLD, m, n, M, N, 0, d_nnz_A, 0, o_nnz_A, &Mat_A);
>> MatSetFromOptions(Mat_A);
>> 
>> for (int i=rstart_A[rank]; i<rend_A[rank]; i++) {
>> mat_value=val_A[i]+PETSC_i*0.0;
>> MatSetValue(Mat_A, i_A[i],j_A[i],mat_value,ADD_VALUES);
>> }
>> 
>> MatAssemblyBegin(Mat_A, MAT_FINAL_ASSEMBLY);
>> MatAssemblyEnd(Mat_A, MAT_FINAL_ASSEMBLY);
>> 
>> PetscViewer viewer;
>> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"Mat_A",&viewer);
>> MatView(Mat_A,viewer);
>> PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_MATLAB);
> 
> Did you intend to use the MATLAB format?  If so, MatView needs to be
> called while the format is pushed, so swap the two lines above.
> 
>> PetscViewerPopFormat(viewer);
>> PetscViewerDestroy(&viewer);


From adener at anl.gov  Fri Jan 25 10:29:11 2019
From: adener at anl.gov (Dener, Alp)
Date: Fri, 25 Jan 2019 16:29:11 +0000
Subject: [petsc-users] [tao] General L1,L2 optimization
In-Reply-To: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>
References: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>
Message-ID: <128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>

Hi Hansol,

We don?t have a Levenberg-Marquardt method available, and if the PETSc/TAO manual says otherwise, that may be misleading. Let me know where you saw that and I can take a look and fix it.

In the meantime, if you want to solve a least-squares problem, the master branch of PETSc on Bitbucket has a bound-constrained regularized Gauss-Newton (TAOBRGN) method available. The only available regularization right now is an L2 proximal point Tikhonov regularizer. There are ongoing efforts to support an L1 regularizer, and also the ability for users to define their own, but these have not made it into the master branch yet. We?re working on it and should be in for the next major PETSc release in the Spring.

If you?d like to use that method, you need to set the Tao type to TAOBRGN and then go through the TaoSetResidualRoutine() and TaoSetJacobianResidualRoutine() interfaces to define your problem.

In general, you can use other TAO algorithms (e.g.: BNLS, BQNLS, etc.) with your own regularization term by embedding it into the objective, gradient and Hessian (if applicable) evaluation callbacks. The caveat is that your regularizer needs to be C1 continuous for first-order methods and C2 continuous for second order methods. This typically limits you to L2-norm regularizers. There is no support yet for L1-norm regularizers, but as I said, we?re working on it right now and it should be available in a couple of months.

Hope that helps,
??
Alp Dener
Argonne National Laboratory
https://www.anl.gov/profile/alp-dener



On Jan 24, 2019, at 2:57 PM, David via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

Hi. I was wondering whether there was some kind of general consensus about

the currently-best-implemented L1, L2 norm regularization for petsc/tao
that has been implemented.

Naively, I would shoot for Levenberg-Marquardt for some kind of random
matrix, or even generic

finite-difference stencil problem. (but it seems like LM is yet to be
implemented, but only on petsc manual pdf?)

Or perhaps, of the implemented ones, LMVM seems to work well, at least
on my local machine.

In any due case, I would highly appreciate the input and opinion about
these matters.


Thanks.



Hansol Suh,

PhD Student


Georgia Institute of Technology


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/0e81aeba/attachment.html>

From jed at jedbrown.org  Fri Jan 25 11:23:19 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 25 Jan 2019 10:23:19 -0700
Subject: [petsc-users] [tao] General L1,L2 optimization
In-Reply-To: <128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>
References: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>
	<128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>
Message-ID: <87k1isd52w.fsf@jedbrown.org>

I don't think Tristan is looking for users just yet, but he has an LM
that he's using for bundle adjustment (comparing with Ceres) here.  We
hope to merge this once it's better tested.

  https://bitbucket.org/tristankonolie/petsc/commits/all

"Dener, Alp via petsc-users" <petsc-users at mcs.anl.gov> writes:

> Hi Hansol,
>
> We don?t have a Levenberg-Marquardt method available, and if the PETSc/TAO manual says otherwise, that may be misleading. Let me know where you saw that and I can take a look and fix it.
>
> In the meantime, if you want to solve a least-squares problem, the master branch of PETSc on Bitbucket has a bound-constrained regularized Gauss-Newton (TAOBRGN) method available. The only available regularization right now is an L2 proximal point Tikhonov regularizer. There are ongoing efforts to support an L1 regularizer, and also the ability for users to define their own, but these have not made it into the master branch yet. We?re working on it and should be in for the next major PETSc release in the Spring.
>
> If you?d like to use that method, you need to set the Tao type to TAOBRGN and then go through the TaoSetResidualRoutine() and TaoSetJacobianResidualRoutine() interfaces to define your problem.
>
> In general, you can use other TAO algorithms (e.g.: BNLS, BQNLS, etc.) with your own regularization term by embedding it into the objective, gradient and Hessian (if applicable) evaluation callbacks. The caveat is that your regularizer needs to be C1 continuous for first-order methods and C2 continuous for second order methods. This typically limits you to L2-norm regularizers. There is no support yet for L1-norm regularizers, but as I said, we?re working on it right now and it should be available in a couple of months.
>
> Hope that helps,
> ??
> Alp Dener
> Argonne National Laboratory
> https://www.anl.gov/profile/alp-dener
>
>
>
> On Jan 24, 2019, at 2:57 PM, David via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Hi. I was wondering whether there was some kind of general consensus about
>
> the currently-best-implemented L1, L2 norm regularization for petsc/tao
> that has been implemented.
>
> Naively, I would shoot for Levenberg-Marquardt for some kind of random
> matrix, or even generic
>
> finite-difference stencil problem. (but it seems like LM is yet to be
> implemented, but only on petsc manual pdf?)
>
> Or perhaps, of the implemented ones, LMVM seems to work well, at least
> on my local machine.
>
> In any due case, I would highly appreciate the input and opinion about
> these matters.
>
>
> Thanks.
>
>
>
> Hansol Suh,
>
> PhD Student
>
>
> Georgia Institute of Technology

From evanum at gmail.com  Fri Jan 25 11:31:13 2019
From: evanum at gmail.com (Evan Um)
Date: Fri, 25 Jan 2019 09:31:13 -0800
Subject: [petsc-users] Printing parallel matrix
In-Reply-To: <F9FF5D27-C487-4BD4-AFF9-424D088E40EE@anl.gov>
References: <CAP1yThSYvjmZpvPZYN7Ef_P4+jp9o8VS6qDQj71foju2yv4oBA@mail.gmail.com>
	<87r2d3idqn.fsf@jedbrown.org>
	<F9FF5D27-C487-4BD4-AFF9-424D088E40EE@anl.gov>
Message-ID: <CAP1yThTiNz7rsNmzNj=AmR4RfecZ-L9cvw=+1Kb5rt5z2ijruA@mail.gmail.com>

Hi Barry,

Yes, it was very large sparse matrix (million rows). I had no problem when
the binary format was used. I think that I need to change my preference.
Thank you very much for your comments and help!

Best regards,
Evan


On Thu, Jan 24, 2019 at 7:58 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   Is it a very large matrix? The ASCII output is intended for only
> smallish matrices and may take a lot of time for very large matrices. For
> large matrices we recommend the binary format which is much faster for
> large matrices and can handle tens of millions of rows. You can read it
> into Python or Matlab with utilities we provide.
>
>     Barry
>
>
> > On Jan 23, 2019, at 9:40 AM, Jed Brown via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Evan Um via petsc-users <petsc-users at mcs.anl.gov> writes:
> >
> >> Dear PETSC users,
> >>
> >> I try to verify a matrix by printing a PETSC matrix and comparing its
> >> elements with reference values. Below is my test code.
> >>
> >> It works well when a single process is used. The output file is created
> >> quickly. In contrast, when multiple processes (>2) are used, printing
> >> matrix is stuck (the parallel matrix is assembled very quickly owing to
> >> memory preallocation). The output file just prints two lines below. No
> >> element is printed.
> >>
> >> Mat Object: 2 MPI processes
> >>  type: mpiaij
> >>
> >> I assume that printing matrix is also designed for a parallel matrix.
> Does
> >> this suggest that my parallel matrix includes any errors? Otherwise,
> does
> >> this work only for a serial matrix?
> >>
> >> Thanks for reading this question.
> >>
> >> Regards,
> >> Evan
> >>
> >> MatCreateAIJ(PETSC_COMM_WORLD, m, n, M, N, 0, d_nnz_A, 0, o_nnz_A,
> &Mat_A);
> >> MatSetFromOptions(Mat_A);
> >>
> >> for (int i=rstart_A[rank]; i<rend_A[rank]; i++) {
> >> mat_value=val_A[i]+PETSC_i*0.0;
> >> MatSetValue(Mat_A, i_A[i],j_A[i],mat_value,ADD_VALUES);
> >> }
> >>
> >> MatAssemblyBegin(Mat_A, MAT_FINAL_ASSEMBLY);
> >> MatAssemblyEnd(Mat_A, MAT_FINAL_ASSEMBLY);
> >>
> >> PetscViewer viewer;
> >> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"Mat_A",&viewer);
> >> MatView(Mat_A,viewer);
> >> PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_MATLAB);
> >
> > Did you intend to use the MATLAB format?  If so, MatView needs to be
> > called while the format is pushed, so swap the two lines above.
> >
> >> PetscViewerPopFormat(viewer);
> >> PetscViewerDestroy(&viewer);
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/b3494621/attachment-0001.html>

From jczhang at mcs.anl.gov  Fri Jan 25 11:32:10 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Fri, 25 Jan 2019 17:32:10 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C4C4Ues6Ssu1SCPKYkPAdnMGE7n0itQDavbH1jKzTzCrA@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
	<CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>
	<CAE3k7C4C4Ues6Ssu1SCPKYkPAdnMGE7n0itQDavbH1jKzTzCrA@mail.gmail.com>
Message-ID: <CA+MQGp-_Ek=MAyr8P=T_-QVXSfN+ONjDrTdBdEeTBY-mLWLkAA@mail.gmail.com>

Hi, Sal Am,
 I did some testes with your matrix and vector. It is a complex matrix with N=4.7M and nnz=417M. Firstly, I tested on a machine with 36 cores and 128GB memory on each compute node. I tried with direct solver and iterative solver but both failed. For example, with 36 ranks on one compute node, I got
[9]PETSC ERROR: [9] SuperLU_DIST:pzgssvx line 465 /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[9]PETSC ERROR: [9] MatLUFactorNumeric_SuperLU_DIST line 314 /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
  With 16 nodes, 576 ranks. I got
SUPERLU_MALLOC fails for GAstore->rnzval[] at line 240 in file /blues/gpfs/home/jczhang/petsc/bdw-dbg-complex/externalpackages/git.superlu_dist/SRC/pzutil.c

  Next, I moved to another single-node machine with 1.5TB memory. It did not fail this time. It ran overnight and is still doing superlu. Using the top command, I found at peak, it consumed almost all memory. At stable period, with 36 ranks, each rank consumed about 20GB memory. When I changed to iterative solvers with -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable, I did not meet errors seen on the smaller memory machine. But the residual did not converge.
  So, I think the errors you met were simply out of memory error either in superlu or in petsc.  If you have machines with large memory, you can try it on. Otherwise, I let other petsc developers suggest better iterative solvers to you.
  Thanks.

--Junchao Zhang


On Wed, Jan 23, 2019 at 2:52 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Sorry it took long had to see if I could shrink down the problem files from 50GB to something smaller (now ~10GB).
Can you compress your matrix and upload it to google drive, so we can try to reproduce the error.

How I ran the problem: mpiexec valgrind --tool=memcheck --suppressions=$HOME/valgrind/valgrind-openmpi.supp -q --num-callers=20 --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

here is the link to matrix A and vector b: https://drive.google.com/drive/folders/16YQPTK6TfXC6pV5RMdJ9g7X-ZiqbvwU8?usp=sharing

I redid the problem (twice) by trying to solve a 1M finite elements problem corresponding to ~ 4M n and 417M nnz matrix elements on the login shell which has ~550GB mem, but it failed. First time it failed because of bus error, second time it said killed. I have attached valgrind file from both runs.

OpenMPI is not my favorite. You need to use a suppressions file to get rid of all of that noise. Here is one:

Thanks I have been using it, but sometimes I still see same amount of errors.



On Fri, Jan 18, 2019 at 3:12 AM Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:
Usually when I meet a SEGV error, I will run it again with a parallel debugger like DDT and wait for it to segfault, and then examine the stack trace to see what is wrong.
Can you compress your matrix and upload it to google drive, so we can try to reproduce the error.
--Junchao Zhang


On Thu, Jan 17, 2019 at 10:44 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I did two runs, one with the SuperLU_dist and one with bcgs using jacobi, attached are the results of one of the reports from valgrind on one random processor (out of the 128 files).

DS = direct solver
IS = iterative  solver

There is an awful lot of errors.

how I initiated the two runs:
mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce -ksp_monitor -log_view

mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

Thank you

On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
1) Running out of memory

2) You passed an invalid array
I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script, i.e. using 300GB/node, a total of 1200GB memory, using 4 nodes and 32 processors per node (128 processors in total).
I am not sure what would constitute an invalid array or how I can check that. I am using the same procedure as when dealing with the smaller matrix. i.e. Generate matrix A and vector b using FEM software then convert the matrix and vector using a python script ready for petsc. read in petsc and calculate.

Are you running with 64-bit ints here?
Yes I have it configured petsc with  --with-64-bit-indices and debugging mode, which this was run on.

It sounds like you have enough memory, but the fact that is runs for smaller problems makes me suspicious. It
could still be a memory overwrite. Can you either

a) Run under valgrind

or

b) Run under the debugger and get a stack trace

 ?

   Thanks,

    Matt

On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
SuperLU_dist supports 64-bit ints. Are you not running in parallel?
I will try that, although I think solving the real problem (later on if I can get this to work) with 30 million finite elements might be a problem for SuperLU_dist. so it is better to get an iterative solver to work with first.

1) Try using    -build_twosided allreduce    on this run
How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce -ksp_monitor -log_view
I have attached the full error output.

You are getting an SEGV on MatSetValues(), so its either

1) Running out of memory

2) You passed an invalid array

2) Is it possible to get something that fails here but we can run. None of our tests show this problem.
I am not how I can do that, but i have added my code which is quite short and should only read and solve the system, the problem arises at larger matrices for example current test case has 6 million finite elements (~2B non-zero numbers and 25M x 25M matrix).

Are you running with 64-bit ints here?

   Matt

On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
The memory requested is an insane number. You may need to use 64 bit integers.
Thanks Mark, I reconfigured it to use 64bit, however in the process it says I can no longer use MUMPS and SuperLU as they are not supported (I see on MUMPS webpage it supports 64int). However it does not exactly solve the problem.

SuperLU_dist supports 64-bit ints. Are you not running in parallel?

This time, it crashes at
[6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
ierr      = PetscMalloc1(bi[pn]+1,&bj);
which allocates local portion of B^T*A.
You may also try to increase number of cores to reduce local matrix size.

So I increased the number of cores to 16 on one node and ran it by :
mpiexec valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
It crashed after reading in the matrix and before starting to solve. The error:

[15]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[3]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR: ------------------------------------------------------------------------
[8]PETSC ERROR: ------------------------------------------------------------------------
[12]PETSC ERROR: ------------------------------------------------------------------------
[12]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------
[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
--------------------------------------------------------------------------
mpiexec noticed that process rank 10 with PID 0 on node r03n01 exited on signal 9 (Killed).

Now I was running this with valgrind as someone had previously suggested and the 16 files created all contain the same type of error:

Okay, its possible that there are bugs in the MPI implementation. So

1) Try using    -build_twosided allreduce    on this run

2) Is it possible to get something that fails here but we can run. None of our tests show this problem.

  Thanks,

     Matt

==25940== Invalid read of size 8
==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:373)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==  Address 0x19f807fc is 12 bytes inside a block of size 16 alloc'd
==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier (mpits.c:371)
==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq (mpits.c:572)
==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
==25940==


On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Fande:
According to this PR https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff

Should we set the scalable algorithm as default?
Sure, we can. But I feel we need do more tests to compare scalable and non-scalable algorithms.
On theory, for small to medium matrices, non-scalable matmatmult() algorithm enables more efficient
data accessing. Andreas optimized scalable implementation. Our non-scalable implementation might have room to be further optimized.
Hong

On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Add option '-mattransposematmult_via scalable'
Hong

On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I saw the following error message in your first email.
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
Probably the matrix is too large. You can try with more compute nodes, for example, use 8 nodes instead of 2, and see what happens.

--Junchao Zhang


On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Using a larger problem set with 2B non-zero elements and a matrix of 25M x 25M I get the following error:
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR:       is given.
[4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
[4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
[4]PETSC ERROR: [4] MatTransposeMatMult line 9950 /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
[4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
[4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r02g03 by vef002 Fri Jan 11 09:13:23 2019
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-superlu --download-fblaslapack=1 --download-cmake
[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

Using Valgrind on only one of the valgrind files the following error was written:

==9053== Invalid read of size 4
==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays (aij.c:4445)
==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ (matmatmult.c:790)
==9053==    by 0x5D106F8: MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:1186)
==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
==9053==  Address 0x8386997f4 is not stack'd, malloc'd or (recently) free'd
==9053==


On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com<mailto:tempohoper at gmail.com>> wrote:
Thank you Dave,

I reconfigured PETSc with valgrind and debugging mode, I ran the code again with the following options:
mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type gamg -log_view
(as on the petsc website you linked)

It finished solving using the iterative solver, but the resulting valgrind.log.%p files (all 8 corresponding to each processor) are all empty. And it took a whooping ~15hours, for what used to take ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is the log_view.


On Thu, Jan 10, 2019 at 8:59 AM Dave May <dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>> wrote:


On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I am not sure what is exactly is wrong as the error changes slightly every time I run it (without changing the parameters).

This likely implies that you have a memory error in your code (a memory leak would not cause this behaviour).
I strongly suggest you make sure your code is free of memory errors.
You can do this using valgrind. See here

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

for an explanation of how to use valgrind.

I have attached the first two run's errors and my code.

Is there a memory leak somewhere? I have tried running it with -malloc_dump, but not getting anything printed out, however, when run with -log_view I see that Viewer is created 4 times, but destroyed 3 times. The way I see it, I have destroyed it where I see I no longer have use for it so not sure if I am wrong. Could this be the reason why it keeps crashing? It crashes as soon as it reads the matrix, before entering the solving mode (I have a print statement before solving starts that never prints).

how I run it in the job script on 2 node with 32 processors using the clusters OpenMPI.

mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg -ksp_converged_reason -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view

the matrix:
2 122 821 366 (non-zero elements)
25 947 279 x 25 947 279

Thanks and all the best


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/37f18440/attachment-0001.html>

From chen2018 at purdue.edu  Fri Jan 25 11:41:53 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Fri, 25 Jan 2019 17:41:53 +0000
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <CAMYG4GmaP=NvNOo0iuRQsxQkF1Eoqcb3tkkWTwMK6z-SRdqJEg@mail.gmail.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
	<DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gko_pmpYfBi9qewWmrS_je3xdcygvDny7zhUJxcA0hnRg@mail.gmail.com>
	<DM5PR22MB02657A04F2A8BFFCA246E494A89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<DM5PR22MB02650936E5F7715A61472E7CA89A0@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CAMYG4GmaP=NvNOo0iuRQsxQkF1Eoqcb3tkkWTwMK6z-SRdqJEg@mail.gmail.com>
Message-ID: <DM5PR22MB0265D97319617CB77C07C3CFA89B0@DM5PR22MB0265.namprd22.prod.outlook.com>

Matt,

      Thanks so much for your help.Now PETSC is successfully linked to our code. But in the program I use PETSC, I still define some variable with the fortran defaulted type:

     PetscErrorCode ierr
     PetscMPIInt rank,size
     integer,allocatable  ::info(:)
     real(wp), allocatable :: Ae(:,:), auxRHSe(:)
     integer, allocatable      :: idx(:)
     PC               prec
     Vec              xvec,bvec,uvec

Then I try to use the following code to assemble my matrix but somehow it does not work:
      do geq=Istart,Iend-1,1
            call ptSystem%getElementalMAT( geq+1, Ae, auxRHSe, idx)      ! subroutine for elemental matrix
            ne=size(idx)
            call matdisp(Ae)
            call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
            call VecSetValues(bvec,ne,idx,auxRHSe,INSERT_VALUES,ierr)
     end do

It seems I can't use size function here. Why is this and how can I get its size?

My professor also said I may generate a static library of PETSC and add it to our code so that I will have less trouble linking problems. Is this doable?


Thanks again and all the best


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, January 24, 2019 10:43 AM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov
Subject: Re: Fw: [petsc-users]Integrate PETSC with existing Fortran

On Thu, Jan 24, 2019 at 10:11 AM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

However, I changed the makefile,but I still received the following error:

Your includes are still not right:

master $:/PETSc3/petsc/petsc-dev$ make getincludedirs
make getincludedirs
-I/PETSc3/petsc/petsc-dev/include -I/PETSc3/petsc/petsc-dev/arch-master-debug/include -I/PETSc3/petsc/petsc-dev/arch-master-debug/include/eigen3 -I/opt/X11/include -I/PETSc3/petsc/include

You are missing the one for the $PETSC_ARCH include directory.

   Matt

HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.

________________________________
From: Yaxiong Chen
Sent: Thursday, January 24, 2019 9:50 AM
To: Matthew Knepley
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran


I see. This is because I have one .c file in the bunch of files.

!==============================================================

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
This command in makefile will use CC  instead of FC since I change f90 to F90.


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Thursday, January 24, 2019 5:54 AM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Wed, Jan 23, 2019 at 2:31 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

    I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90. But I still received the following error:

HiDACX/obj/debug/optimal_mechanical_part.o
mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90 -o HiDACX/obj/debug/optimal_mechanical_part.o
error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


Is there anything wrong with the flag?

You are compiling a Fortran file with mpicc, instead of mpif90?

   Matt


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, January 23, 2019 1:30 PM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,


I tried to modify the structure of present code and use KSP in the main program(optimal_mechanical_part).  Now the makefile is as following :

#================================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
HiDAC_Utils/HiDAC_Utils.f90 \
HiDAC_Material/HiDAC_baseMat.f90 \
HiDAC_Material/HiDAC_isoElasticMat.f90 \
HiDAC_Material/HiDAC_thermoMat.f90 \
HiDAC_Material/HiDAC_Material.f90 \
HiDAC_Approximation/HiDAC_node.f90 \
HiDAC_Approximation/HiDAC_nodeNet.f90 \
HiDAC_Approximation/HiDAC_baseApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90 \
HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Domain/HiDAC_enrichment.f90 \
HiDAC_Domain/HiDAC_composedDomain.f90 \
HiDAC_Domain/HiDAC_Domain.f90 \
HiDAC_Quadrature/HiDAC_quadType.f90 \
HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
HiDAC_Quadrature/HiDAC_Quadrature.f90 \
HiDAC_Analysis/HiDAC_BC.f90 \
HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
HiDAC_Analysis/HiDAC_elastostatics.f90 \
HiDAC_Analysis/HiDAC_heatConduction.f90 \
HiDAC_Analysis/HiDAC_phaseTransition.f90 \
HiDAC_Analysis/HiDAC_fracture.f90 \
HiDAC_Analysis/HiDAC_lineenrich.f90 \
HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
HiDAC_Analysis/HiDAC_Analysis.f90 \
HiDAC_Solver/HiDAC_baseSolver.f90 \
HiDAC_Solver/HiDAC_LapackSolver.f90 \
HiDAC_Solver/HiDAC_MumpsSolver.f90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $(OBJS_DIR)
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}

$(LIB): $(OBJS)
echo $(OBJS)
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)optimal_mechanical_part.o: \
optimal_mechanical_part.f90 \
$(OBJS_DIR)read_iges_file.o \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Domain.o \
$(OBJS_DIR)HiDAC_Material.o \
$(OBJS_DIR)HiDAC_Quadrature.o \
$(OBJS_DIR)HiDAC_Analysis.o \
$(OBJS_DIR)HiDAC_Utils.o

$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

$(OBJS_DIR)dxf_reader.o: \
dxf_reader.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)parse.o: \
parse.f90 \
$(OBJS_DIR)nrb.o

$(OBJS_DIR)map.o: \
map.f90 \
$(OBJS_DIR)nrb.o

<Other dependencies >
# $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
# mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
# -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
# -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90 \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
# /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
#    This part is removed

endif

#============================Makefile finished ===============================



and the main program starts like this:


!============================  Main program starts=========================

program main
  !******************************************************!
  ! Import all HiDAC library
  !******************************************************!
  include 'mpif.h'
#include <petsc/finclude/petscksp.h>
  use petscksp
  use read_iges_file
  use HiDAC_Approximation
  use HiDAC_Domain
  use HiDAC_Material
  use HiDAC_Analysis
  use HiDAC_Utils
  use HiDAC_quadType

  implicit none
!=================================================================

I have not call the function in PETSC yet. Just add the KSP library in my main program. But I received  the following error when compiling:


HiDACX/obj/debug/optimal_mechanical_part.o
mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90 -o HiDACX/obj/debug/optimal_mechanical_part.o
/Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:

 #if defined (PETSC_HAVE_MPIUNI)
  1~~~~~~~~~~~~
Fatal Error: petscconf.h: No such file or directory
compilation terminated.
make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
make: *** [debug] Error 2


How can I solve this problem?

You named the file

  optimal_mechanical_part.f90

but it has to be

  optimal_mechanical_part.F90

in order for the compiler to preprocess it.

   Matt


Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 2:21 PM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Do not drop the Cc.


I have hundreds .F90 files to compile as well as an .c file to compile in this project. The present makefile is more complex than the PETSC makefile. Do you mean I should use the PETSC makefile as a template and add all the dependencies in it?

If you want to do it by hand you can, its just more error prone and verbose.


My present Makefile is like the following(Red parts are news added related to Petsc). Is this the right direction? Since a static pattern is defined while the compiling for PetscSolver is a little different. I guess I should list it as a specific case instead of using the pattern.

I will comment inline.


# ============================================================================
# Name        : Makefile
# Author      : Tao Song
# Version     : 2.1
# Copyright   : GNU 2.1
# Description : Makefile for HiDACX
# ============================================================================

OUT_DIR := HiDACX/

## Define mode
ifndef MODE

.PHONY: debug release

debug: export MODE := debug
release: export MODE := release
debug release:
@$(MAKE) $@

clean:
rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/

else

## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
SRCS_DIR := $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
FSRCS := \
HiDAC_Utils/HiDAC_parameter.f90 \
HiDAC_Utils/HiDAC_dataStructure.f90 \
......
<other .f90 files>
HiDAC_Domain/HiDAC_primitive.f90 \
HiDAC_Solver/HiDAC_PetscSolver.F90 \
HiDAC_Solver/HiDAC_Solver.f90 \
IGES/read_iges_file.f90 \
IGES/nrb.f90 \
IGES/parse.f90 \
IGES/map.f90 \
DXF/dxf_reader.f90 \

TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/

CSRCS := \
HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \

SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))

OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
LIB_DIR := $(OUT_DIR)lib/$(MODE)/
EXE_DIR := $(OUT_DIR)bin/$(MODE)/
EXT_DIR := /usr/local/

#petsc lib
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

You do not need 'rules' if you are doing things by hand. Although it is easier to just use the builtin rule for building source.


#========================

OBJS := $(addprefix $(OBJS_DIR), \
$(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
$(patsubst %.c, %.o, $(notdir $(CSRCS))))

LIB = $(LIB_DIR)libhidac.a

VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)

FC := mpif90
CC := mpicc
AR := ar rcs

FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs -ftest-coverage
CCFLAGS := -Dqh_QHpointer -ansi
LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}

FIDIR := \
-I$(EXT_DIR)include \
-I$(PETSC_DIR)/include
CIDIR := \
-I$(EXT_DIR)include/libqhull

# sequence does matter
EXTLIBS := \
-L$(EXT_DIR)lib \
-ldmumps -lmumps_common \
-lparmetis -lmetis -lpord -lscalapack \
-llapack -lblas -lqhullstatic_p \
-lpetsc


.PHONY: debug release all clean

# Handle different mode using target-specific variable
debug: FCFLAGS += -Wall -fcheck=bounds
debug: CCFLAGS += -Wall
debug: LFLAGS += -Wall -fcheck=bounds

release: FCFLAGS += -O2
release: CCFLAGS += -O2
release: LFLAGS += -O2

debug release: $(EXE_DIR)optimal_mechanical_part.hdc

## Rules
.SUFFIXES: .hdc
# Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter! Lib later!
$(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
@mkdir -p $(EXE_DIR)
echo $OBJS_DIR
$(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)

$(LIB): $(OBJS)
echo $OBJS
@mkdir -p $(LIB_DIR)
$(AR) $@ $^

$(OBJS_DIR)%.o:
@mkdir -p $(OBJS_DIR)
@echo $@
$(if $(findstring .f90, $<), \
$(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
$(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)

## Dependencies of files
$(OBJS_DIR)read_iges_file.o: \
read_iges_file.f90 \
$(OBJS_DIR)HiDAC_Approximation.o \
$(OBJS_DIR)HiDAC_Utils.o \
$(OBJS_DIR)nrb.o \
$(OBJS_DIR)parse.o \
$(OBJS_DIR)map.o \
$(OBJS_DIR)dxf_reader.o

$(OBJS_DIR)test_main.o: \
test_main.f90 \
$(OBJS_DIR)read_iges_file.o

<.... Other dependencies>

$ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90 $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
-I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
-I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90 \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o \
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o \
${PETSC_KSP_LIB}

You can do this, but to be certain you should should use

  ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}

along with your flags (-J, etc.). Notice that you do not need the *.o files here since you are only compiling.

You can simplify this by includeing 'rules' and just putting the $ (OBJS_DIR)HiDAC_PetscSolver.o as a
dependence for your link.

  Thanks,

    Matt

endif



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, January 18, 2019 11:57 AM
To: Yaxiong Chen; PETSc
Subject: Re: [petsc-users]Integrate PETSC with existing Fortran

On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Hi Matt,

Now I have some problem with makefile.I want to integrate PETSC with a previous FEM Fortran code. To do that, I add a module named PetscSolver. It starts as the following:

It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the PETSc makefiles to avoid problems.

  Matt


!===================================================================

  module HiDAC_PetscSolver
#include <petsc/finclude/petscksp.h>
  use petscksp
  use HiDAC_Utils
  use HiDAC_baseSolver
    implicit none
  private
  type, extends(Solver), public  :: PetscSolver
    private
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
  contains
     procedure :: initialize => initialize_PetscSolver
     procedure :: assemble => assemble_PetscSolver
     procedure :: solve => solve_PetscSolver
     procedure :: postprocess => postprocess_PetscSolver
     procedure :: finalize => finalize_PetscSolver
     procedure :: getID
  end type PetscSolver
......
<Functions>
end module HiDAC_PetscSolver
!===================================================================
However, it did not compile successfully. It gave me the error:

 #include <petsc/finclude/petscksp.h>
  1
Warning: Illegal preprocessor directive
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:

      Vec              xvec,bvec,uvec
     1
Error: Unclassifiable statement at (1)
/Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:

      Mat              Amat
All function and type related to Petsc are not defined.

I must have made a very silly mistake. Can you show me how to fix it?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/37e70529/attachment-0001.html>

From tisaac at cc.gatech.edu  Fri Jan 25 12:11:02 2019
From: tisaac at cc.gatech.edu (Isaac, Tobin G)
Date: Fri, 25 Jan 2019 18:11:02 +0000
Subject: [petsc-users] [tao] General L1,L2 optimization
In-Reply-To: <128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>
References: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>
	<128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>
Message-ID: <20190125181057.co5frhtghicybgfv@gatech.edu>


On Fri, Jan 25, 2019 at 04:29:11PM +0000, Dener, Alp via petsc-users wrote:
> Hi Hansol,
> 
> We don?t have a Levenberg-Marquardt method available, and if the PETSc/TAO manual says otherwise, that may be misleading. Let me know where you saw that and I can take a look and fix it.
> 
> In the meantime, if you want to solve a least-squares problem, the master branch of PETSc on Bitbucket has a bound-constrained regularized Gauss-Newton (TAOBRGN) method available. The only available regularization right now is an L2 proximal point Tikhonov regularizer. There are ongoing efforts to support an L1 regularizer, and also the ability for users to define their own, but these have not made it into the master branch yet. We?re working on it and should be in for the next major PETSc release in the Spring.
> 
> If you?d like to use that method, you need to set the Tao type to TAOBRGN and then go through the TaoSetResidualRoutine() and TaoSetJacobianResidualRoutine() interfaces to define your problem.
> 
> In general, you can use other TAO algorithms (e.g.: BNLS, BQNLS, etc.) with your own regularization term by embedding it into the objective, gradient and Hessian (if applicable) evaluation callbacks. The caveat is that your regularizer needs to be C1 continuous for first-order methods and C2 continuous for second order methods. This typically limits you to L2-norm regularizers. There is no support yet for L1-norm regularizers, but as I said, we?re working on it right now and it should be available in a couple of months.

Is there a feature branch yet where we could see what is being
prepared?

While I think it is nice as a convenience for Tao to support adding l1
regularization for people, my two cents is that this should call on a
deeper structure that supports separable objectives.

For example, an implementation of total variation regularization would
apply l1 to some operator applied to the control variables, and
instead of creating a complete TV implementation in Tao, a user should
be able to express "This part (misfit) is smooth" and "This part is
not, but it is convex".

  Toby


> 
> Hope that helps,
> ??
> Alp Dener
> Argonne National Laboratory
> https://www.anl.gov/profile/alp-dener
> 
> 
> 
> On Jan 24, 2019, at 2:57 PM, David via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> 
> Hi. I was wondering whether there was some kind of general consensus about
> 
> the currently-best-implemented L1, L2 norm regularization for petsc/tao
> that has been implemented.
> 
> Naively, I would shoot for Levenberg-Marquardt for some kind of random
> matrix, or even generic
> 
> finite-difference stencil problem. (but it seems like LM is yet to be
> implemented, but only on petsc manual pdf?)
> 
> Or perhaps, of the implemented ones, LMVM seems to work well, at least
> on my local machine.
> 
> In any due case, I would highly appreciate the input and opinion about
> these matters.
> 
> 
> Thanks.
> 
> 
> 
> Hansol Suh,
> 
> PhD Student
> 
> 
> Georgia Institute of Technology
> 
> 

From tisaac at cc.gatech.edu  Fri Jan 25 12:26:12 2019
From: tisaac at cc.gatech.edu (Isaac, Tobin G)
Date: Fri, 25 Jan 2019 18:26:12 +0000
Subject: [petsc-users] Tao big picture?
Message-ID: <20190125182609.g7j5pf6akhsbtu7z@gatech.edu>


Hi guys,

It's clear from the recent petsc-users exchange that you have big
changes in store for Tao, and that a bunch of us are also interested
in this development.  I know there's a big roadmap meeting coming up:
is there a design document somewhere that my student and I (and
others thinking about Tao development) can see?

Thanks,
  Toby

From adener at anl.gov  Fri Jan 25 12:47:55 2019
From: adener at anl.gov (Dener, Alp)
Date: Fri, 25 Jan 2019 18:47:55 +0000
Subject: [petsc-users] [tao] General L1,L2 optimization
In-Reply-To: <20190125181057.co5frhtghicybgfv@gatech.edu>
References: <b77534f6-842f-4368-7a17-a866ed9e6fa6@gatech.edu>
	<128D9FB9-A0BA-469E-94C1-66FB9ED0ECCE@anl.gov>
	<20190125181057.co5frhtghicybgfv@gatech.edu>
Message-ID: <FA50157C-EB35-4AE0-8495-913C3C1E8FC8@anl.gov>

Hi Toby,

On Jan 25, 2019, at 12:11 PM, Isaac, Tobin G <tisaac at cc.gatech.edu<mailto:tisaac at cc.gatech.edu>> wrote:


On Fri, Jan 25, 2019 at 04:29:11PM +0000, Dener, Alp via petsc-users wrote:
Hi Hansol,

We don?t have a Levenberg-Marquardt method available, and if the PETSc/TAO manual says otherwise, that may be misleading. Let me know where you saw that and I can take a look and fix it.

In the meantime, if you want to solve a least-squares problem, the master branch of PETSc on Bitbucket has a bound-constrained regularized Gauss-Newton (TAOBRGN) method available. The only available regularization right now is an L2 proximal point Tikhonov regularizer. There are ongoing efforts to support an L1 regularizer, and also the ability for users to define their own, but these have not made it into the master branch yet. We?re working on it and should be in for the next major PETSc release in the Spring.

If you?d like to use that method, you need to set the Tao type to TAOBRGN and then go through the TaoSetResidualRoutine() and TaoSetJacobianResidualRoutine() interfaces to define your problem.

In general, you can use other TAO algorithms (e.g.: BNLS, BQNLS, etc.) with your own regularization term by embedding it into the objective, gradient and Hessian (if applicable) evaluation callbacks. The caveat is that your regularizer needs to be C1 continuous for first-order methods and C2 continuous for second order methods. This typically limits you to L2-norm regularizers. There is no support yet for L1-norm regularizers, but as I said, we?re working on it right now and it should be available in a couple of months.

Is there a feature branch yet where we could see what is being
prepared?

Xiang has been working on it here: https://bitbucket.org/petsc/petsc/branch/xhuang/tao-feature-L1

Currently it?s a complete replacement of the L2 regularizer in BRGN, but we will merge and combine with the old regularizer, and make some interface changes to support user-provided regularizers before it?s ready for a pull request.


While I think it is nice as a convenience for Tao to support adding l1
regularization for people, my two cents is that this should call on a
deeper structure that supports separable objectives.

The Gauss-Newton code (BRGN) is a very simple wrapper around the existing Newton Line-Search (BNLS) algorithm. It constructs the problem from the provided residual and Jacobian, tacks on an L2 regularizer, and feeds it into BNLS underneath.

I?m not intimately familiar with the L1 regularizer that Xiang has been working on for some tomography problems but I?m reasonably sure that it does not take advantage of any kind of a structure in the objective function because it?s built on top of the same BRGN infrastructure.

I wasn?t aiming for a very sophisticated least squares solver when I put together BRGN. It?s more of a low-hanging fruit case, where we could write a very small amount of code that saves the user some time in constructing the same Gauss-Newton problem that they could actually have also constructed themselves manually if they defined the FormFunctionAndGradient() and FormHessian() routines appropriately for existing Newton solvers.

I agree that a more sophisticated and serious approach to least-squares problems would benefit from separable objective functions and algorithms that take advantage of structure, but we don?t have any solvers in Tao that do this yet. I personally wouldn?t mind adding other features to BRGN or adding other L-S solvers, but we don?t really have applications or use-cases in front of us so it?s hasn?t risen high up on the priority list. If you have a problem you want to solve or a particular algorithm to add to Tao, we can talk altogether with Todd and see what we have to do with Tao interfaces and other things to support those efforts and collaborate.

Best,
??
Alp Dener
Argonne National Laboratory
https://www.anl.gov/profile/alp-dener


For example, an implementation of total variation regularization would
apply l1 to some operator applied to the control variables, and
instead of creating a complete TV implementation in Tao, a user should
be able to express "This part (misfit) is smooth" and "This part is
not, but it is convex".

 Toby



Hope that helps,
??
Alp Dener
Argonne National Laboratory
https://www.anl.gov/profile/alp-dener



On Jan 24, 2019, at 2:57 PM, David via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:petsc-users at mcs.anl.gov>> wrote:

Hi. I was wondering whether there was some kind of general consensus about

the currently-best-implemented L1, L2 norm regularization for petsc/tao
that has been implemented.

Naively, I would shoot for Levenberg-Marquardt for some kind of random
matrix, or even generic

finite-difference stencil problem. (but it seems like LM is yet to be
implemented, but only on petsc manual pdf?)

Or perhaps, of the implemented ones, LMVM seems to work well, at least
on my local machine.

In any due case, I would highly appreciate the input and opinion about
these matters.


Thanks.



Hansol Suh,

PhD Student


Georgia Institute of Technology

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/b0ae92e9/attachment-0001.html>

From knepley at gmail.com  Fri Jan 25 14:22:35 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Jan 2019 15:22:35 -0500
Subject: [petsc-users] Integrate PETSC with existing Fortran
In-Reply-To: <DM5PR22MB0265D97319617CB77C07C3CFA89B0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265BD5A814C63E48209412BA8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GnWhRCUtC6J38XG8ymrwH+XT+sV9jErSON8NfXnk-Zppw@mail.gmail.com>
	<DM5PR22MB0265FA216F8525FC7B334451A8850@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=nTquVCZV_7w0KMzQuHULOefM0EpM7znnxLY6VzhSHeQ@mail.gmail.com>
	<DM5PR22MB0265AAFE9AD83C2679C54F00A89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4G=AAH0XR_mbyt6Mdu5fzZVSsNiSUNDZpWUqEWvKS_iDJg@mail.gmail.com>
	<DM5PR22MB026589684299948BF50BEA7BA89C0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gn7WNLRKq8Xq6dSSdjHvK0LCNqsF+0C6EaTMsP84UYmXw@mail.gmail.com>
	<DM5PR22MB0265E6ED3C259DD4AE1CD6DBA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmUAc5ikMEQLQqaHiGse5Ekt3_GwhgZJZtKTtSqEQ2ixA@mail.gmail.com>
	<DM5PR22MB02652875EC3F1609F4C05B9AA8990@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4Gko_pmpYfBi9qewWmrS_je3xdcygvDny7zhUJxcA0hnRg@mail.gmail.com>
	<DM5PR22MB02657A04F2A8BFFCA246E494A89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<DM5PR22MB02650936E5F7715A61472E7CA89A0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAMYG4GmaP=NvNOo0iuRQsxQkF1Eoqcb3tkkWTwMK6z-SRdqJEg@mail.gmail.com>
	<DM5PR22MB0265D97319617CB77C07C3CFA89B0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAMYG4Gn8YweNnsCUwt2JXMg5UCW+cVz+oyJsnVTXHWOHzQ24vw@mail.gmail.com>

On Fri, Jan 25, 2019 at 12:41 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> Matt,
>
>       Thanks so much for your help.Now PETSC is successfully linked to our
> code. But in the program I use PETSC, I still define some variable with the
> fortran defaulted type:
>
>      PetscErrorCode ierr
>      PetscMPIInt rank,size
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:)
>      integer, allocatable      :: idx(:)
>      PC               prec
>      Vec              xvec,bvec,uvec
>
> Then I try to use the following code to assemble my matrix but somehow it
> does not work:
>       do geq=Istart,Iend-1,1
>             call ptSystem%getElementalMAT( geq+1, Ae, auxRHSe, idx)      !
> subroutine for elemental matrix
>             ne=size(idx)
>             call matdisp(Ae)
>             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
>             call VecSetValues(bvec,ne,idx,auxRHSe,INSERT_VALUES,ierr)
>      end do
>
> It seems I can't use size function here. Why is this and how can I get its
> size?
>

You are calling 'size', which must be an F90 function, on 'idx' which is a
pure F90 type. What does this have to do with PETSc?
By this I mean, isn't this just a Fortran problem that can be solved by a
Fortran expert? Maybe I am wrong.

> My professor also said I may generate a static library of PETSC and add it
> to our code so that I will have less trouble linking problems. Is this
> doable?
>

It is doable, but it will not make anything easier. Your linking problems
had nothing to do with shared libraries. It
had to do with missing symbols.

  Thanks,

    Matt


> Thanks again and all the best
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, January 24, 2019 10:43 AM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: Fw: [petsc-users]Integrate PETSC with existing Fortran
>
> On Thu, Jan 24, 2019 at 10:11 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> However, I changed the makefile,but I still received the following error:
>
>
> Your includes are still not right:
>
> master $:/PETSc3/petsc/petsc-dev$ make getincludedirs
> make getincludedirs
> -I/PETSc3/petsc/petsc-dev/include
> -I/PETSc3/petsc/petsc-dev/arch-master-debug/include
> -I/PETSc3/petsc/petsc-dev/arch-master-debug/include/eigen3
> -I/opt/X11/include -I/PETSc3/petsc/include
>
> You are missing the one for the $PETSC_ARCH include directory.
>
>    Matt
>
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
>
> ------------------------------
> *From:* Yaxiong Chen
> *Sent:* Thursday, January 24, 2019 9:50 AM
> *To:* Matthew Knepley
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
>
> I see. This is because I have one .c file in the bunch of files.
>
> !==============================================================
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
> This command in makefile will use CC  instead of FC since I change f90 to
> F90.
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, January 24, 2019 5:54 AM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Wed, Jan 23, 2019 at 2:31 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>     I changed optimal_mechanical_part.f90 to  optimal_mechanical_part.F90.
> But I still received the following error:
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpicc -Dqh_QHpointer -ansi -Wall -I/usr/local/include/libqhull -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.F90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
> error: invalid value 'f95-cpp-input' in '-x f95-cpp-input'
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> Is there anything wrong with the flag?
>
>
> You are compiling a Fortran file with mpicc, instead of mpif90?
>
>    Matt
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Wednesday, January 23, 2019 1:30 PM
> *To:* Yaxiong Chen
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Wed, Jan 23, 2019 at 1:27 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> I tried to modify the structure of present code and use KSP in the main
> program(optimal_mechanical_part).  Now the makefile is as following :
>
>
> #================================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathFunction.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_mathType.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_tensor.f90 \
> HiDAC_Utils/HiDAC_Math/HiDAC_Math.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_IGES.f90 \
> HiDAC_Utils/HiDAC_Port/IR_Precision.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Pack_Data.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_Base64.f90 \
> HiDAC_Utils/HiDAC_Port/Lib_VTK_IO.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_VTK.f90 \
> HiDAC_Utils/HiDAC_Port/HiDAC_Port.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_F.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_BezierHull.f90 \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_DistanceField.f90 \
> HiDAC_Utils/HiDAC_Utils.f90 \
> HiDAC_Material/HiDAC_baseMat.f90 \
> HiDAC_Material/HiDAC_isoElasticMat.f90 \
> HiDAC_Material/HiDAC_thermoMat.f90 \
> HiDAC_Material/HiDAC_Material.f90 \
> HiDAC_Approximation/HiDAC_node.f90 \
> HiDAC_Approximation/HiDAC_nodeNet.f90 \
> HiDAC_Approximation/HiDAC_baseApproximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_knot.f90 \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_BezierApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_NURBSApproximation.f90
> \
> HiDAC_Approximation/HiDAC_Approximation_NURBS/HiDAC_THBApproximation.f90 \
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Domain/HiDAC_enrichment.f90 \
> HiDAC_Domain/HiDAC_composedDomain.f90 \
> HiDAC_Domain/HiDAC_Domain.f90 \
> HiDAC_Quadrature/HiDAC_quadType.f90 \
> HiDAC_Quadrature/HiDAC_directQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_adaptiveQuadrature.f90 \
> HiDAC_Quadrature/HiDAC_Quadrature.f90 \
> HiDAC_Analysis/HiDAC_BC.f90 \
> HiDAC_Analysis/HiDAC_baseAnalysis.f90 \
> HiDAC_Analysis/HiDAC_elastostatics.f90 \
> HiDAC_Analysis/HiDAC_heatConduction.f90 \
> HiDAC_Analysis/HiDAC_phaseTransition.f90 \
> HiDAC_Analysis/HiDAC_fracture.f90 \
> HiDAC_Analysis/HiDAC_lineenrich.f90 \
> HiDAC_Analysis/HiDAC_sensitivityAnalysis.f90 \
> HiDAC_Analysis/HiDAC_Analysis.f90 \
> HiDAC_Solver/HiDAC_baseSolver.f90 \
> HiDAC_Solver/HiDAC_LapackSolver.f90 \
> HiDAC_Solver/HiDAC_MumpsSolver.f90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $(OBJS_DIR)
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS) ${PETSC_KSP_LIB}
>
> $(LIB): $(OBJS)
> echo $(OBJS)
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)optimal_mechanical_part.o: \
> optimal_mechanical_part.f90 \
> $(OBJS_DIR)read_iges_file.o \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Domain.o \
> $(OBJS_DIR)HiDAC_Material.o \
> $(OBJS_DIR)HiDAC_Quadrature.o \
> $(OBJS_DIR)HiDAC_Analysis.o \
> $(OBJS_DIR)HiDAC_Utils.o
>
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> $(OBJS_DIR)dxf_reader.o: \
> dxf_reader.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)parse.o: \
> parse.f90 \
> $(OBJS_DIR)nrb.o
>
> $(OBJS_DIR)map.o: \
> map.f90 \
> $(OBJS_DIR)nrb.o
>
> <Other dependencies >
> # $(OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.f90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> # mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> # -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> # -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> # /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> #    This part is removed
>
> endif
>
> #============================Makefile
> finished ===============================
>
>
>
> and the main program starts like this:
>
>
> !============================  Main program starts=========================
>
> program main
>   !******************************************************!
>   ! Import all HiDAC library
>   !******************************************************!
>   include 'mpif.h'
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use read_iges_file
>   use HiDAC_Approximation
>   use HiDAC_Domain
>   use HiDAC_Material
>   use HiDAC_Analysis
>   use HiDAC_Utils
>   use HiDAC_quadType
>
>   implicit none
> !====================*=============*================================
>
> I have not call the function in PETSC yet. Just add the KSP library in my
> main program. But I received  the following error when compiling:
>
>
> HiDACX/obj/debug/optimal_mechanical_part.o
> mpif90 -cpp -JHiDACX/obj/debug/  -Wall -fcheck=bounds -I/usr/local/include
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include -c
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/HiDAC_Test/optimal_mechanical_part.f90
> -o HiDACX/obj/debug/optimal_mechanical_part.o
>
> /Users/yaxiong/Downloads/petsc-3.9.4/include/petsc/finclude/petscsys.h:13:2:
>
>  #if defined (PETSC_HAVE_MPIUNI)
>   1~~~~~~~~~~~~
> Fatal Error: petscconf.h: No such file or directory
> compilation terminated.
> make[1]: *** [HiDACX/obj/debug/optimal_mechanical_part.o] Error 1
> make: *** [debug] Error 2
>
> How can I solve this problem?
>
>
> You named the file
>
>   optimal_mechanical_part.f90
>
> but it has to be
>
>   optimal_mechanical_part.F90
>
> in order for the compiler to preprocess it.
>
>    Matt
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 2:21 PM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 2:06 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
>
> Do not drop the Cc.
>
>
> I have hundreds .F90 files to compile as well as an .c file to compile in
> this project. The present makefile is more complex than the PETSC makefile.
> Do you mean I should use the PETSC makefile as a template and add all the
> dependencies in it?
>
>
> If you want to do it by hand you can, its just more error prone and
> verbose.
>
>
> My present Makefile is like the following(Red parts are news added related
> to Petsc). Is this the right direction? Since a static pattern is defined
> while the compiling for PetscSolver is a little different. I guess I should
> list it as a specific case instead of using the pattern.
>
>
> I will comment inline.
>
>
> #
> ============================================================================
> # Name        : Makefile
> # Author      : Tao Song
> # Version     : 2.1
> # Copyright   : GNU 2.1
> # Description : Makefile for HiDACX
> #
> ============================================================================
>
> OUT_DIR := HiDACX/
>
> ## Define mode
> ifndef MODE
>
> .PHONY: debug release
>
> debug: export MODE := debug
> release: export MODE := release
> debug release:
> @$(MAKE) $@
>
> clean:
> rm -rf $(OUT_DIR)obj/ $(OUT_DIR)lib/ $(OUT_DIR)bin/
>
> else
>
> ## SRCS = $(shell find ../../MyGit/Hidac_Fortran_Lib/ "*.f90")
> SRCS_DIR :=
> $(HOME)/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_Petsc/Hidac_Fortran_Lib/
> FSRCS := \
> HiDAC_Utils/HiDAC_parameter.f90 \
> HiDAC_Utils/HiDAC_dataStructure.f90 \
> ......
> <other .f90 files>
> HiDAC_Domain/HiDAC_primitive.f90 \
> HiDAC_Solver/HiDAC_PetscSolver.F90 \
> HiDAC_Solver/HiDAC_Solver.f90 \
> IGES/read_iges_file.f90 \
> IGES/nrb.f90 \
> IGES/parse.f90 \
> IGES/map.f90 \
> DXF/dxf_reader.f90 \
>
> TSRCS_DIR := $(SRCS_DIR)HiDAC_Test/
>
> CSRCS := \
> HiDAC_Utils/HiDAC_DistanceField/HiDAC_QhullInterface_C.c \
>
> SRCS := $(addprefix $(SRCS_DIR), $(FSRCS) $(CSRCS))
>
> OBJS_DIR := $(OUT_DIR)obj/$(MODE)/
> LIB_DIR := $(OUT_DIR)lib/$(MODE)/
> EXE_DIR := $(OUT_DIR)bin/$(MODE)/
> EXT_DIR := /usr/local/
>
> #petsc lib
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
>
>
> You do not need 'rules' if you are doing things by hand. Although it is
> easier to just use the builtin rule for building source.
>
>
>
> #========================
>
> OBJS := $(addprefix $(OBJS_DIR), \
> $(patsubst %.f90, %.o, $(notdir $(FSRCS))) \
> $(patsubst %.c, %.o, $(notdir $(CSRCS))))
>
> LIB = $(LIB_DIR)libhidac.a
>
> VPATH := $(sort $(dir $(SRCS))) $(TSRCS_DIR)
>
> FC := mpif90
> CC := mpicc
> AR := ar rcs
>
> FCFLAGS := -cpp -J$(OBJS_DIR) # ${PETSC_KSP_LIB}# -fprofile-arcs
> -ftest-coverage
> CCFLAGS := -Dqh_QHpointer -ansi
> LFLAGS := -Wl,-rpath,$(EXT_DIR)lib, ${PETSC_KSP_LIB}
>
> FIDIR := \
> -I$(EXT_DIR)include \
> -I$(PETSC_DIR)/include
> CIDIR := \
> -I$(EXT_DIR)include/libqhull
>
> # sequence does matter
> EXTLIBS := \
> -L$(EXT_DIR)lib \
> -ldmumps -lmumps_common \
> -lparmetis -lmetis -lpord -lscalapack \
> -llapack -lblas -lqhullstatic_p \
> -lpetsc
>
>
> .PHONY: debug release all clean
>
> # Handle different mode using target-specific variable
> debug: FCFLAGS += -Wall -fcheck=bounds
> debug: CCFLAGS += -Wall
> debug: LFLAGS += -Wall -fcheck=bounds
>
> release: FCFLAGS += -O2
> release: CCFLAGS += -O2
> release: LFLAGS += -O2
>
> debug release: $(EXE_DIR)optimal_mechanical_part.hdc
>
> ## Rules
> .SUFFIXES: .hdc
> # Note the sequence of $(OBJS_DIR)%.o $(LIB) and $(EXTLIBS) does matter!
> Lib later!
> $(EXE_DIR)%.hdc: $(OBJS_DIR)%.o $(LIB)
> @mkdir -p $(EXE_DIR)
> echo $OBJS_DIR
> $(FC) $(LFLAGS) -o $@ $^ $(EXTLIBS)
>
> $(LIB): $(OBJS)
> echo $OBJS
> @mkdir -p $(LIB_DIR)
> $(AR) $@ $^
>
> $(OBJS_DIR)%.o:
> @mkdir -p $(OBJS_DIR)
> @echo $@
> $(if $(findstring .f90, $<), \
> $(FC) $(FCFLAGS) $(FIDIR) -c $< -o $@, \
> $(CC) $(CCFLAGS) $(CIDIR) -c $< -o $@)
>
> ## Dependencies of files
> $(OBJS_DIR)read_iges_file.o: \
> read_iges_file.f90 \
> $(OBJS_DIR)HiDAC_Approximation.o \
> $(OBJS_DIR)HiDAC_Utils.o \
> $(OBJS_DIR)nrb.o \
> $(OBJS_DIR)parse.o \
> $(OBJS_DIR)map.o \
> $(OBJS_DIR)dxf_reader.o
>
> $(OBJS_DIR)test_main.o: \
> test_main.f90 \
> $(OBJS_DIR)read_iges_file.o
>
> <.... Other dependencies>
>
> $ (OBJS_DIR)HiDAC_PetscSolver.o : HiDAC_PetscSolver.F90
> $(OBJS_DIR)HiDAC_baseSolver.o $(OBJS_DIR)HiDAC_Utils.o
> mpif90 -cpp -JHiDACX/obj/debug/ -Wall -ffree-line-length-0
> -Wno-unused-dummy-argument -g
> -I/Users/yaxiong/Downloads/petsc-3.9.4/include \
> -I/Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/include \
> -I/opt/X11/include -o $(OBJS_DIR)HiDAC_PetscSolver.o \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.F90
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_baseSolver.o
> \
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDACX/obj/debug/HiDAC_Utils.o
> \
> ${PETSC_KSP_LIB}
>
>
> You can do this, but to be certain you should should use
>
>   ${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS}
>
> along with your flags (-J, etc.). Notice that you do not need the *.o
> files here since you are only compiling.
>
> You can simplify this by includeing 'rules' and just putting the $
> (OBJS_DIR)HiDAC_PetscSolver.o as a
> dependence for your link.
>
>   Thanks,
>
>     Matt
>
>
> endif
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Friday, January 18, 2019 11:57 AM
> *To:* Yaxiong Chen; PETSc
> *Subject:* Re: [petsc-users]Integrate PETSC with existing Fortran
>
> On Fri, Jan 18, 2019 at 11:37 AM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Hi Matt,
>
> Now I have some problem with makefile.I want to integrate PETSC with a
> previous FEM Fortran code. To do that, I add a module named PetscSolver. It
> starts as the following:
>
>
> It needs to have suffix .F or .F90 so the preprocessor is invoked. Use the
> PETSc makefiles to avoid problems.
>
>   Matt
>
>
> !===================================================================
>
>   module HiDAC_PetscSolver
> #include <petsc/finclude/petscksp.h>
>   use petscksp
>   use HiDAC_Utils
>   use HiDAC_baseSolver
>     implicit none
>   private
>   type, extends(Solver), public  :: PetscSolver
>     private
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>   contains
>      procedure :: initialize => initialize_PetscSolver
>      procedure :: assemble => assemble_PetscSolver
>      procedure :: solve => solve_PetscSolver
>      procedure :: postprocess => postprocess_PetscSolver
>      procedure :: finalize => finalize_PetscSolver
>      procedure :: getID
>   end type PetscSolver
> ......
> <Functions>
> end module HiDAC_PetscSolver
> !===================================================================
> However, it did not compile successfully. It gave me the error:
>
>  #include <petsc/finclude/petscksp.h>
>   1
> Warning: Illegal preprocessor directive
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:23:5:
>
>       Vec              xvec,bvec,uvec
>      1
> Error: Unclassifiable statement at (1)
>
> /Users/yaxiong/Documents/HiDAC_Code_Git/HiDAC_Tao_NURBS_PETSC/Hidac_Fortran_Lib/HiDAC_Solver/HiDAC_PetscSolver.f90:24:5:
>
>       Mat              Amat
> All function and type related to Petsc are not defined.
>
> I must have made a very silly mistake. Can you show me how to fix it?
>
> Thanks
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/980da7e0/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Fri Jan 25 16:51:49 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 25 Jan 2019 16:51:49 -0600
Subject: [petsc-users] Question about TSSetRHSJacobian for linear time
 dependent problem
Message-ID: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>

Hi,

If I have a linear time dependent equation I'm trying to solve using TS, I
can use :
TSSetProblemType(ts,TS_LINEAR);
TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian, &appctx);

If the matrix that's being evaluated by YourComputeRHSJacobian is such that
the non-zero structure stays the same and only the diagonal changes with
time, is there a way to optimize the function so that it doesn't create the
whole matrix from scratch each time ?

Naively I can make a dummy matrix and store the copy from t=0 and change
the diagonal at each iteration but that unnecessarily doubles the memory
consumption, is there a better way?


Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/efb8b491/attachment.html>

From hongzhang at anl.gov  Fri Jan 25 19:29:09 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Sat, 26 Jan 2019 01:29:09 +0000
Subject: [petsc-users] Question about TSSetRHSJacobian for linear time
 dependent problem
In-Reply-To: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
References: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
Message-ID: <D122E1ED-B88B-418E-AB31-1C7F087A9A9E@anl.gov>



On Jan 25, 2019, at 4:51 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

Hi,

If I have a linear time dependent equation I'm trying to solve using TS, I can use :
TSSetProblemType(ts,TS_LINEAR);
TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian, &appctx);

If the matrix that's being evaluated by YourComputeRHSJacobian is such that the non-zero structure stays the same and only the diagonal changes with time, is there a way to optimize the function so that it doesn't create the whole matrix from scratch each time ?

The matrix A should be created only once before TSSetRHSJacobian() is called. YourComputeRHSJacobian() does not need to create the whole matrix each time; it updates the values in A and assembles it.

Naively I can make a dummy matrix and store the copy from t=0 and change the diagonal at each iteration but that unnecessarily doubles the memory consumption, is there a better way?

A is the working matrix storing the Jacobian. The matrix passed to YourComputeRHSJacobian() is actually A. There is no need to save an extra copy.

Hong (Mr.)


Thank You,
Sajid Ali
Applied Physics
Northwestern University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190126/b18ddc91/attachment.html>

From gohardoust at gmail.com  Fri Jan 25 09:44:39 2019
From: gohardoust at gmail.com (Mohammad Gohardoust)
Date: Fri, 25 Jan 2019 15:44:39 +0000
Subject: [petsc-users] Slow linear solver via MUMPS
Message-ID: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>

Hi,

I am trying to modify a "pure MPI" code for solving water movement equation
in soils which employs KSP iterative solvers. This code gets really slow in
the hpc I am testing it as I increase the number of calculating nodes (each
node has 28 cores) even from 1 to 2. I went for implementing "MPI-OpenMP"
solutions like MUMPS. I did this inside the petsc by:

KSPSetType(ksp, KSPPREONLY);
PCSetType(pc, PCLU);
PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);
KSPSolve(ksp, ...

and I run it through:

export OMP_NUM_THREADS=16 && mpirun -n 2 ~/Programs/my_programs

The code is working (in my own PC) but it is too slow (maybe about 50 times
slower). Since I am not an expert, I like to know is this what I should
expect from MUMPS!?

Thanks,
Mohammad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190125/7dfc9877/attachment.html>

From knepley at gmail.com  Sat Jan 26 06:51:07 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 26 Jan 2019 07:51:07 -0500
Subject: [petsc-users] Slow linear solver via MUMPS
In-Reply-To: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
References: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
Message-ID: <CAMYG4Gm+2XXj8f6NioGefBU61CXs+fO=eMYZBwYdHHNTm9_T_A@mail.gmail.com>

On Fri, Jan 25, 2019 at 9:06 PM Mohammad Gohardoust via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I am trying to modify a "pure MPI" code for solving water movement
> equation in soils which employs KSP iterative solvers. This code gets
> really slow in the hpc I am testing it as I increase the number of
> calculating nodes (each node has 28 cores) even from 1 to 2. I went for
> implementing "MPI-OpenMP" solutions like MUMPS. I did this inside the petsc
> by:
>

In my opinion, this is not a great strategy. You should figure out what is
happening with MUMPS first. Second, you
should consider not using LU because it is not that scalable. First, when
you run in parallel, are you having MUMPS
calculate the ordering in parallel? Second, it sounds like you are running
on a single node. Do not expect to get much
speedup with either processes or threads here because all your operations
are limited by memory bandwidth, not
compute power, and the node has a fixed bandwidth.

  Thanks,

    Matt


> KSPSetType(ksp, KSPPREONLY);
> PCSetType(pc, PCLU);
> PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);
> KSPSolve(ksp, ...
>
> and I run it through:
>
> export OMP_NUM_THREADS=16 && mpirun -n 2 ~/Programs/my_programs
>
> The code is working (in my own PC) but it is too slow (maybe about 50
> times slower). Since I am not an expert, I like to know is this what I
> should expect from MUMPS!?
>
> Thanks,
> Mohammad
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190126/5d3bed46/attachment.html>

From jczhang at mcs.anl.gov  Sat Jan 26 20:20:29 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Sun, 27 Jan 2019 02:20:29 +0000
Subject: [petsc-users] Slow linear solver via MUMPS
In-Reply-To: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
References: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
Message-ID: <CA+MQGp9=PgGJb2XQOzWPyErVLymwKQLYhdtN-cHaNuzArM=0nw@mail.gmail.com>




On Fri, Jan 25, 2019 at 8:07 PM Mohammad Gohardoust via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi,

I am trying to modify a "pure MPI" code for solving water movement equation in soils which employs KSP iterative solvers. This code gets really slow in the hpc I am testing it as I increase the number of calculating nodes (each node has 28 cores) even from 1 to 2. I went for implementing "MPI-OpenMP" solutions like MUMPS. I did this inside the petsc by:

KSPSetType(ksp, KSPPREONLY);
PCSetType(pc, PCLU);
PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);
KSPSolve(ksp, ...

and I run it through:

export OMP_NUM_THREADS=16 && mpirun -n 2 ~/Programs/my_programs

On some cases, I saw multithreaded MUMPS could improve its performance about 30%. I guess something was wrong in your tests.  You need to compile MUMPS with OpenMP support. If you installed MUMPS through PETSc, you need petsc configure option --with-openmp=1. In addition, you need a multithreaded BLAS. You can do it through --download-openblas

But first of all, you should add --log_view to report your performance results.
The code is working (in my own PC) but it is too slow (maybe about 50 times slower). Since I am not an expert, I like to know is this what I should expect from MUMPS!?

Thanks,
Mohammad

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190127/93662874/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan 27 16:24:38 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 27 Jan 2019 22:24:38 +0000
Subject: [petsc-users] Question about TSSetRHSJacobian for linear time
 dependent problem
In-Reply-To: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
References: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
Message-ID: <8B9ADF66-36B6-4BB5-B34F-B0CB0A348D0D@anl.gov>



> On Jan 25, 2019, at 4:51 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi, 
> 
> If I have a linear time dependent equation I'm trying to solve using TS, I can use :
> TSSetProblemType(ts,TS_LINEAR);
> TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
> TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian, &appctx); 
> 
> If the matrix that's being evaluated by YourComputeRHSJacobian is such that the non-zero structure stays the same and only the diagonal changes with time, is there a way to optimize the function so that it doesn't create the whole matrix from scratch each time ? 

If it is a linear PDE u_t = A u  then how can A change with time? It sounds like it really isn't a linear problem?

   Barry

> 
> Naively I can make a dummy matrix and store the copy from t=0 and change the diagonal at each iteration but that unnecessarily doubles the memory consumption, is there a better way? 
> 
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University


From sajidsyed2021 at u.northwestern.edu  Sun Jan 27 16:26:47 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Sun, 27 Jan 2019 16:26:47 -0600
Subject: [petsc-users] Question about TSSetRHSJacobian for linear time
 dependent problem
In-Reply-To: <8B9ADF66-36B6-4BB5-B34F-B0CB0A348D0D@anl.gov>
References: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
	<8B9ADF66-36B6-4BB5-B34F-B0CB0A348D0D@anl.gov>
Message-ID: <CAOGsD9jt_SZGw6Yz5rRHikpsmE54hASJiO8QZKCyAiq4Xe7u0A@mail.gmail.com>

The form is u_t = A(t)u.

On Sun, Jan 27, 2019 at 4:24 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 25, 2019, at 4:51 PM, Sajid Ali via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hi,
> >
> > If I have a linear time dependent equation I'm trying to solve using TS,
> I can use :
> > TSSetProblemType(ts,TS_LINEAR);
> > TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
> > TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian, &appctx);
> >
> > If the matrix that's being evaluated by YourComputeRHSJacobian is such
> that the non-zero structure stays the same and only the diagonal changes
> with time, is there a way to optimize the function so that it doesn't
> create the whole matrix from scratch each time ?
>
> If it is a linear PDE u_t = A u  then how can A change with time? It
> sounds like it really isn't a linear problem?
>
>    Barry
>
> >
> > Naively I can make a dummy matrix and store the copy from t=0 and change
> the diagonal at each iteration but that unnecessarily doubles the memory
> consumption, is there a better way?
> >
> >
> > Thank You,
> > Sajid Ali
> > Applied Physics
> > Northwestern University
>
>

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190127/b5ef80f3/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan 27 16:41:02 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 27 Jan 2019 22:41:02 +0000
Subject: [petsc-users] Question about TSSetRHSJacobian for linear time
 dependent problem
In-Reply-To: <CAOGsD9jt_SZGw6Yz5rRHikpsmE54hASJiO8QZKCyAiq4Xe7u0A@mail.gmail.com>
References: <CAOGsD9iFSHWczmGcE7AEOhsPx=iB=b2MS7irHtG+V5_Z7p-ySg@mail.gmail.com>
	<8B9ADF66-36B6-4BB5-B34F-B0CB0A348D0D@anl.gov>
	<CAOGsD9jt_SZGw6Yz5rRHikpsmE54hASJiO8QZKCyAiq4Xe7u0A@mail.gmail.com>
Message-ID: <4AB1F1BC-A967-41B2-B731-7A1931C7A77A@mcs.anl.gov>



> On Jan 27, 2019, at 4:26 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu> wrote:
> 
> The form is u_t = A(t)u. 

   Got it. 

   You can call TSRHSJacobianSetReuse() after you create the TS object and then each time your compute jacobian is called it will be passed the same matrix as before. Then you can use MatDiagonalSet() to change only the diagonal values. Good luck.

   Barry

> 
> On Sun, Jan 27, 2019 at 4:24 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
> > On Jan 25, 2019, at 4:51 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> >
> > Hi,
> >
> > If I have a linear time dependent equation I'm trying to solve using TS, I can use :
> > TSSetProblemType(ts,TS_LINEAR);
> > TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
> > TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian, &appctx);
> >
> > If the matrix that's being evaluated by YourComputeRHSJacobian is such that the non-zero structure stays the same and only the diagonal changes with time, is there a way to optimize the function so that it doesn't create the whole matrix from scratch each time ?
> 
> If it is a linear PDE u_t = A u  then how can A change with time? It sounds like it really isn't a linear problem?
> 
>    Barry
> 
> >
> > Naively I can make a dummy matrix and store the copy from t=0 and change the diagonal at each iteration but that unnecessarily doubles the memory consumption, is there a better way?
> >
> >
> > Thank You,
> > Sajid Ali
> > Applied Physics
> > Northwestern University
> 
> 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University


From overholt at capesim.com  Mon Jan 28 08:26:30 2019
From: overholt at capesim.com (Matthew Overholt)
Date: Mon, 28 Jan 2019 09:26:30 -0500
Subject: [petsc-users] Slow linear solver via MUMPS
In-Reply-To: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
References: <CAPqpBe8snygVL+W_bhhDM8aurpOAE_ksFkwc5eLrYg=wpfuvEA@mail.gmail.com>
Message-ID: <CAMSLWku9fhiUr-ov6YU42Rg_5hxveh_008-TKNcKE_5YOwQO_w@mail.gmail.com>

Hi Mohammad,

We tried the same thing for our finite element heat transfer code, and
experimented with both MUMPS and MKL's Cluster PARDISO for about a year,
and were very disappointed with how they scaled.

Give the full PETSc PCG solver with the ILU(0) preconditioner a try (pure
MPI, no hybrid MPI-OpenMP).  We found that it scales very well over two or
more nodes, and even though it is slower than MKL PARDISO on a single node,
its speedup is so much better over multiple MPI ranks that it quickly
overtakes the speed of the direct solvers.

ierr = KSPSetType(ksp, KSPCG);                       // And stick with the
default ILU preconditioner

The interconnect we've been using is 25 Gbps Ethernet, which is standard on
the AWS EC2 cloud.

Matt Overholt

On Fri, Jan 25, 2019 at 10:44 AM Mohammad Gohardoust via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I am trying to modify a "pure MPI" code for solving water movement
> equation in soils which employs KSP iterative solvers. This code gets
> really slow in the hpc I am testing it as I increase the number of
> calculating nodes (each node has 28 cores) even from 1 to 2. I went for
> implementing "MPI-OpenMP" solutions like MUMPS. I did this inside the petsc
> by:
>
> KSPSetType(ksp, KSPPREONLY);
> PCSetType(pc, PCLU);
> PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);
> KSPSolve(ksp, ...
>
> and I run it through:
>
> export OMP_NUM_THREADS=16 && mpirun -n 2 ~/Programs/my_programs
>
> The code is working (in my own PC) but it is too slow (maybe about 50
> times slower). Since I am not an expert, I like to know is this what I
> should expect from MUMPS!?
>
> Thanks,
> Mohammad
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/c3c039c8/attachment.html>

From nahmad16 at ku.edu.tr  Mon Jan 28 12:48:56 2019
From: nahmad16 at ku.edu.tr (Najeeb Ahmad)
Date: Mon, 28 Jan 2019 21:48:56 +0300
Subject: [petsc-users] mat_type MATSEQAIJCUSPARSE not recognized
Message-ID: <CAJJ73m1Fw+Fq3=1RmoxGt6Vt=9LQeZq5oFY-fdM1qtqunAAGGA@mail.gmail.com>

Dear all,

I am trying to use PETSc with GPU. I have followed instructions laid down
at https://www.mcs.anl.gov/petsc/features/gpus.html. When I run my code, it
doesn't recognize the mat_type MATSEQAIJCUSPARSE. Here is the log:

mpiexec -n 1 ./GMRES_PETSc -fin ../MM2PETScMatConverter/HV15R.bin
-ksp_monitor -pc_type ilu -pc_factor_levels 0 -ksp_gmres_restart 200
-ksp_gmres_preallocate -vec_type VECSEQCUDA -mat_type MATSEQAIJCUSPARSE
Processes: 1
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
http://www.mcs.anl.gov/petsc/documentation/installation.html#external
[0]PETSC ERROR: Unknown Mat type given: MATSEQAIJCUSPARSE
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
GIT Date: 2019-01-25 22:25:25 -0600
[0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
--with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
[0]PETSC ERROR: #1 MatSetType() line 61 in
/home/ubuntu/petsc/src/mat/interface/matreg.c
[0]PETSC ERROR: #2 MatSetFromOptions() line 230 in
/home/ubuntu/petsc/src/mat/utils/gcreate.c
[0]PETSC ERROR: #3 main() line 48 in
/home/ubuntu/koc-aramco/src_GPU/GMRES_PETSc/main.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -fin ../MM2PETScMatConverter/HV15R.bin
[0]PETSC ERROR: -ksp_gmres_preallocate
[0]PETSC ERROR: -ksp_gmres_restart 200
[0]PETSC ERROR: -ksp_monitor
[0]PETSC ERROR: -mat_type MATSEQAIJCUSPARSE
[0]PETSC ERROR: -pc_factor_levels 0
[0]PETSC ERROR: -pc_type ilu
[0]PETSC ERROR: -vec_type VECSEQCUDA
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0
[cli_0]: readline failed
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PetscError line 352
/home/ubuntu/petsc/src/sys/error/err.c
[0]PETSC ERROR: [0] MatSetType line 42
/home/ubuntu/petsc/src/mat/interface/matreg.c
[0]PETSC ERROR: [0] MatSetFromOptions line 214
/home/ubuntu/petsc/src/mat/utils/gcreate.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
GIT Date: 2019-01-25 22:25:25 -0600
[0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
--with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
[0]PETSC ERROR: #4 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 13 Broken Pipe: Likely while reading
or writing to a socket
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PetscError line 352
/home/ubuntu/petsc/src/sys/error/err.c
[0]PETSC ERROR: [0] MatSetType line 42
/home/ubuntu/petsc/src/mat/interface/matreg.c
[0]PETSC ERROR: [0] MatSetFromOptions line 214
/home/ubuntu/petsc/src/mat/utils/gcreate.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
GIT Date: 2019-01-25 22:25:25 -0600
[0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
--with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
[0]PETSC ERROR: #5 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=59
:
system msg for write_line failure : Broken pipe


Any help will be greatly appreciated.

Regards,

-- 
*Najeeb Ahmad*


*Research and Teaching Assistant*
*PARallel and MultiCORE Computing Laboratory (ParCoreLab) *

*Computer Science and Engineering*
*Ko? University, Istanbul, Turkey*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/4e78a9ad/attachment.html>

From knepley at gmail.com  Mon Jan 28 13:32:38 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Jan 2019 14:32:38 -0500
Subject: [petsc-users] mat_type MATSEQAIJCUSPARSE not recognized
In-Reply-To: <CAJJ73m1Fw+Fq3=1RmoxGt6Vt=9LQeZq5oFY-fdM1qtqunAAGGA@mail.gmail.com>
References: <CAJJ73m1Fw+Fq3=1RmoxGt6Vt=9LQeZq5oFY-fdM1qtqunAAGGA@mail.gmail.com>
Message-ID: <CAMYG4GnFtBa8JbQxCBHa8Xn9=EDfVRsWYC0r0DvEOd4A1rsitQ@mail.gmail.com>

Send your configure.log

  Thanks,

    Matt

On Mon, Jan 28, 2019 at 2:05 PM Najeeb Ahmad via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
> Dear all,
>
> I am trying to use PETSc with GPU. I have followed instructions laid down
> at https://www.mcs.anl.gov/petsc/features/gpus.html. When I run my code,
> it doesn't recognize the mat_type MATSEQAIJCUSPARSE. Here is the log:
>
> mpiexec -n 1 ./GMRES_PETSc -fin ../MM2PETScMatConverter/HV15R.bin
> -ksp_monitor -pc_type ilu -pc_factor_levels 0 -ksp_gmres_restart 200
> -ksp_gmres_preallocate -vec_type VECSEQCUDA -mat_type MATSEQAIJCUSPARSE
> Processes: 1
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
> http://www.mcs.anl.gov/petsc/documentation/installation.html#external
> [0]PETSC ERROR: Unknown Mat type given: MATSEQAIJCUSPARSE
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
> GIT Date: 2019-01-25 22:25:25 -0600
> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
> [0]PETSC ERROR: #1 MatSetType() line 61 in
> /home/ubuntu/petsc/src/mat/interface/matreg.c
> [0]PETSC ERROR: #2 MatSetFromOptions() line 230 in
> /home/ubuntu/petsc/src/mat/utils/gcreate.c
> [0]PETSC ERROR: #3 main() line 48 in
> /home/ubuntu/koc-aramco/src_GPU/GMRES_PETSc/main.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -fin ../MM2PETScMatConverter/HV15R.bin
> [0]PETSC ERROR: -ksp_gmres_preallocate
> [0]PETSC ERROR: -ksp_gmres_restart 200
> [0]PETSC ERROR: -ksp_monitor
> [0]PETSC ERROR: -mat_type MATSEQAIJCUSPARSE
> [0]PETSC ERROR: -pc_factor_levels 0
> [0]PETSC ERROR: -pc_type ilu
> [0]PETSC ERROR: -vec_type VECSEQCUDA
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0
> [cli_0]: readline failed
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PetscError line 352
> /home/ubuntu/petsc/src/sys/error/err.c
> [0]PETSC ERROR: [0] MatSetType line 42
> /home/ubuntu/petsc/src/mat/interface/matreg.c
> [0]PETSC ERROR: [0] MatSetFromOptions line 214
> /home/ubuntu/petsc/src/mat/utils/gcreate.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
> GIT Date: 2019-01-25 22:25:25 -0600
> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
> [0]PETSC ERROR: #4 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 13 Broken Pipe: Likely while reading
> or writing to a socket
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PetscError line 352
> /home/ubuntu/petsc/src/sys/error/err.c
> [0]PETSC ERROR: [0] MatSetType line 42
> /home/ubuntu/petsc/src/mat/interface/matreg.c
> [0]PETSC ERROR: [0] MatSetFromOptions line 214
> /home/ubuntu/petsc/src/mat/utils/gcreate.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
> GIT Date: 2019-01-25 22:25:25 -0600
> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
> [0]PETSC ERROR: #5 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=59
> :
> system msg for write_line failure : Broken pipe
>
>
> Any help will be greatly appreciated.
>
> Regards,
>
> --
> *Najeeb Ahmad*
>
>
> *Research and Teaching Assistant*
> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>
> *Computer Science and Engineering*
> *Ko? University, Istanbul, Turkey*
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/238c7ff1/attachment.html>

From nahmad16 at ku.edu.tr  Mon Jan 28 13:36:27 2019
From: nahmad16 at ku.edu.tr (Najeeb Ahmad)
Date: Mon, 28 Jan 2019 22:36:27 +0300
Subject: [petsc-users] mat_type MATSEQAIJCUSPARSE not recognized
In-Reply-To: <CAMYG4GnFtBa8JbQxCBHa8Xn9=EDfVRsWYC0r0DvEOd4A1rsitQ@mail.gmail.com>
References: <CAJJ73m1Fw+Fq3=1RmoxGt6Vt=9LQeZq5oFY-fdM1qtqunAAGGA@mail.gmail.com>
	<CAMYG4GnFtBa8JbQxCBHa8Xn9=EDfVRsWYC0r0DvEOd4A1rsitQ@mail.gmail.com>
Message-ID: <CAJJ73m2xY+Fe=G+mmiwReNRbjOV9i=shd6+5-HyZO5eDwN99Pw@mail.gmail.com>

Dear Matthew,

Please find attached configure.log

On Mon, Jan 28, 2019 at 10:32 PM Matthew Knepley <knepley at gmail.com> wrote:

> Send your configure.log
>
>   Thanks,
>
>     Matt
>
> On Mon, Jan 28, 2019 at 2:05 PM Najeeb Ahmad via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>
>> Dear all,
>>
>> I am trying to use PETSc with GPU. I have followed instructions laid down
>> at https://www.mcs.anl.gov/petsc/features/gpus.html. When I run my code,
>> it doesn't recognize the mat_type MATSEQAIJCUSPARSE. Here is the log:
>>
>> mpiexec -n 1 ./GMRES_PETSc -fin ../MM2PETScMatConverter/HV15R.bin
>> -ksp_monitor -pc_type ilu -pc_factor_levels 0 -ksp_gmres_restart 200
>> -ksp_gmres_preallocate -vec_type VECSEQCUDA -mat_type MATSEQAIJCUSPARSE
>> Processes: 1
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external
>> [0]PETSC ERROR: Unknown Mat type given: MATSEQAIJCUSPARSE
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
>> GIT Date: 2019-01-25 22:25:25 -0600
>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>> [0]PETSC ERROR: #1 MatSetType() line 61 in
>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>> [0]PETSC ERROR: #2 MatSetFromOptions() line 230 in
>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>> [0]PETSC ERROR: #3 main() line 48 in
>> /home/ubuntu/koc-aramco/src_GPU/GMRES_PETSc/main.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -fin ../MM2PETScMatConverter/HV15R.bin
>> [0]PETSC ERROR: -ksp_gmres_preallocate
>> [0]PETSC ERROR: -ksp_gmres_restart 200
>> [0]PETSC ERROR: -ksp_monitor
>> [0]PETSC ERROR: -mat_type MATSEQAIJCUSPARSE
>> [0]PETSC ERROR: -pc_factor_levels 0
>> [0]PETSC ERROR: -pc_type ilu
>> [0]PETSC ERROR: -vec_type VECSEQCUDA
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0
>> [cli_0]: readline failed
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>> batch system) has told this process to end
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
>> X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] PetscError line 352
>> /home/ubuntu/petsc/src/sys/error/err.c
>> [0]PETSC ERROR: [0] MatSetType line 42
>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>> [0]PETSC ERROR: [0] MatSetFromOptions line 214
>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
>> GIT Date: 2019-01-25 22:25:25 -0600
>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>> [0]PETSC ERROR: #4 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 13 Broken Pipe: Likely while reading
>> or writing to a socket
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
>> X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] PetscError line 352
>> /home/ubuntu/petsc/src/sys/error/err.c
>> [0]PETSC ERROR: [0] MatSetType line 42
>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>> [0]PETSC ERROR: [0] MatSetFromOptions line 214
>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1369-g58fe69bfc8
>> GIT Date: 2019-01-25 22:25:25 -0600
>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>> [0]PETSC ERROR: #5 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>> [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=59
>> :
>> system msg for write_line failure : Broken pipe
>>
>>
>> Any help will be greatly appreciated.
>>
>> Regards,
>>
>> --
>> *Najeeb Ahmad*
>>
>>
>> *Research and Teaching Assistant*
>> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>>
>> *Computer Science and Engineering*
>> *Ko? University, Istanbul, Turkey*
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
*Najeeb Ahmad*


*Research and Teaching Assistant*
*PARallel and MultiCORE Computing Laboratory (ParCoreLab) *

*Computer Science and Engineering*
*Ko? University, Istanbul, Turkey*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/c4e0340f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 9204425 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/c4e0340f/attachment-0001.bin>

From knepley at gmail.com  Mon Jan 28 13:42:27 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Jan 2019 14:42:27 -0500
Subject: [petsc-users] mat_type MATSEQAIJCUSPARSE not recognized
In-Reply-To: <CAJJ73m2xY+Fe=G+mmiwReNRbjOV9i=shd6+5-HyZO5eDwN99Pw@mail.gmail.com>
References: <CAJJ73m1Fw+Fq3=1RmoxGt6Vt=9LQeZq5oFY-fdM1qtqunAAGGA@mail.gmail.com>
	<CAMYG4GnFtBa8JbQxCBHa8Xn9=EDfVRsWYC0r0DvEOd4A1rsitQ@mail.gmail.com>
	<CAJJ73m2xY+Fe=G+mmiwReNRbjOV9i=shd6+5-HyZO5eDwN99Pw@mail.gmail.com>
Message-ID: <CAMYG4Gmt5ajh5N1VT8UFNCwoQWdg40=BOiBES5y0sChqmKfjbw@mail.gmail.com>

On Mon, Jan 28, 2019 at 2:36 PM Najeeb Ahmad <nahmad16 at ku.edu.tr> wrote:

> Dear Matthew,
>
> Please find attached configure.log
>
> On Mon, Jan 28, 2019 at 10:32 PM Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> Send your configure.log
>>
>>   Thanks,
>>
>>     Matt
>>
>> On Mon, Jan 28, 2019 at 2:05 PM Najeeb Ahmad via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>>
>>> Dear all,
>>>
>>> I am trying to use PETSc with GPU. I have followed instructions laid
>>> down at https://www.mcs.anl.gov/petsc/features/gpus.html. When I run my
>>> code, it doesn't recognize the mat_type MATSEQAIJCUSPARSE. Here is the log:
>>>
>>> mpiexec -n 1 ./GMRES_PETSc -fin ../MM2PETScMatConverter/HV15R.bin
>>> -ksp_monitor -pc_type ilu -pc_factor_levels 0 -ksp_gmres_restart 200
>>> -ksp_gmres_preallocate -vec_type VECSEQCUDA -mat_type MATSEQAIJCUSPARSE
>>>
>>
I was not looking closely enough last time. The options are

  -vec_type seqcuda -mat_type mpiaijcusparse

  Thanks,

    Matt


> Processes: 1
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing
>>> package:
>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external
>>> [0]PETSC ERROR: Unknown Mat type given: MATSEQAIJCUSPARSE
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.10.3-1369-g58fe69bfc8  GIT Date: 2019-01-25 22:25:25 -0600
>>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>>> [0]PETSC ERROR: #1 MatSetType() line 61 in
>>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>>> [0]PETSC ERROR: #2 MatSetFromOptions() line 230 in
>>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>>> [0]PETSC ERROR: #3 main() line 48 in
>>> /home/ubuntu/koc-aramco/src_GPU/GMRES_PETSc/main.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -fin ../MM2PETScMatConverter/HV15R.bin
>>> [0]PETSC ERROR: -ksp_gmres_preallocate
>>> [0]PETSC ERROR: -ksp_gmres_restart 200
>>> [0]PETSC ERROR: -ksp_monitor
>>> [0]PETSC ERROR: -mat_type MATSEQAIJCUSPARSE
>>> [0]PETSC ERROR: -pc_factor_levels 0
>>> [0]PETSC ERROR: -pc_type ilu
>>> [0]PETSC ERROR: -vec_type VECSEQCUDA
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0
>>> [cli_0]: readline failed
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>>> batch system) has told this process to end
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] PetscError line 352
>>> /home/ubuntu/petsc/src/sys/error/err.c
>>> [0]PETSC ERROR: [0] MatSetType line 42
>>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>>> [0]PETSC ERROR: [0] MatSetFromOptions line 214
>>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.10.3-1369-g58fe69bfc8  GIT Date: 2019-01-25 22:25:25 -0600
>>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>>> [0]PETSC ERROR: #4 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 13 Broken Pipe: Likely while
>>> reading or writing to a socket
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>> OS X to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] PetscError line 352
>>> /home/ubuntu/petsc/src/sys/error/err.c
>>> [0]PETSC ERROR: [0] MatSetType line 42
>>> /home/ubuntu/petsc/src/mat/interface/matreg.c
>>> [0]PETSC ERROR: [0] MatSetFromOptions line 214
>>> /home/ubuntu/petsc/src/mat/utils/gcreate.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.10.3-1369-g58fe69bfc8  GIT Date: 2019-01-25 22:25:25 -0600
>>> [0]PETSC ERROR: ./GMRES_PETSc on a arch-linux2-c-debug named
>>> ip-172-31-86-226 by ubuntu Mon Jan 28 18:10:34 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --download-mpich
>>> --with-cuda=1 --with-cuda-dir=/usr/local/cuda-10.0/ --download-fblaslapack=1
>>> [0]PETSC ERROR: #5 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>> [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=59
>>> :
>>> system msg for write_line failure : Broken pipe
>>>
>>>
>>> Any help will be greatly appreciated.
>>>
>>> Regards,
>>>
>>> --
>>> *Najeeb Ahmad*
>>>
>>>
>>> *Research and Teaching Assistant*
>>> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>>>
>>> *Computer Science and Engineering*
>>> *Ko? University, Istanbul, Turkey*
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> *Najeeb Ahmad*
>
>
> *Research and Teaching Assistant*
> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>
> *Computer Science and Engineering*
> *Ko? University, Istanbul, Turkey*
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190128/b8c36ec5/attachment.html>

From tempohoper at gmail.com  Tue Jan 29 08:05:48 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 29 Jan 2019 14:05:48 +0000
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CA+MQGp-_Ek=MAyr8P=T_-QVXSfN+ONjDrTdBdEeTBY-mLWLkAA@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
	<CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>
	<CAE3k7C4C4Ues6Ssu1SCPKYkPAdnMGE7n0itQDavbH1jKzTzCrA@mail.gmail.com>
	<CA+MQGp-_Ek=MAyr8P=T_-QVXSfN+ONjDrTdBdEeTBY-mLWLkAA@mail.gmail.com>
Message-ID: <CAE3k7C7AuKFpVM=SPM7HUNMFjmR7bKO_FBpkPL-b1_sSWmKUag@mail.gmail.com>

Thank you Zhang for that, I am a bit confused by the terminology maybe, but
when you solved it on 16nodes (each node 128GB) with 576ranks that should
give 2048GB (~2TB) of memory in total right? Does this mean the MPI does
not work for SuperLU_dist? Unfortunately I cannot run the code on one node,
as our nodes are limited to 500GB memory, hence I was hoping I could
utilise several nodes and more memory.

If anyone else has a solution for an iterative solver please do recommend
as I am a bit stuck since so far and the combinations I have tried did not
work.

On Fri, Jan 25, 2019 at 5:32 PM Zhang, Junchao <jczhang at mcs.anl.gov> wrote:

> Hi, Sal Am,
>  I did some testes with your matrix and vector. It is a complex matrix
> with N=4.7M and nnz=417M. Firstly, I tested on a machine with 36 cores and
> 128GB memory on each compute node. I tried with direct solver and iterative
> solver but both failed. For example, with 36 ranks on one compute node, I
> got
>
> [9]PETSC ERROR: [9] SuperLU_DIST:pzgssvx line 465
> /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [9]PETSC ERROR: [9] MatLUFactorNumeric_SuperLU_DIST line 314
> /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>
>   With 16 nodes, 576 ranks. I got
>
> SUPERLU_MALLOC fails for GAstore->rnzval[] at line 240 in file
> /blues/gpfs/home/jczhang/petsc/bdw-dbg-complex/externalpackages/git.superlu_dist/SRC/pzutil.c
>
>   Next, I moved to another single-node machine with 1.5TB memory. It did
> not fail this time. It ran overnight and is still doing superlu. Using the
> top command, I found at peak, it consumed almost all memory. At stable
> period, with 36 ranks, each rank consumed about 20GB memory. When I changed
> to iterative solvers with -ksp_type bcgs -pc_type gamg
> -mattransposematmult_via scalable, I did not meet errors seen on the
> smaller memory machine. But the residual did not converge.
>   So, I think the errors you met were simply out of memory error either in
> superlu or in petsc.  If you have machines with large memory, you can try
> it on. Otherwise, I let other petsc developers suggest better iterative
> solvers to you.
>   Thanks.
>
>
> --Junchao Zhang
>
>
> On Wed, Jan 23, 2019 at 2:52 AM Sal Am <tempohoper at gmail.com> wrote:
>
>> Sorry it took long had to see if I could shrink down the problem files
>> from 50GB to something smaller (now ~10GB).
>>
>>> Can you compress your matrix and upload it to google drive, so we can
>>> try to reproduce the error.
>>>
>>
>> How I ran the problem: mpiexec valgrind --tool=memcheck
>> --suppressions=$HOME/valgrind/valgrind-openmpi.supp -q --num-callers=20
>> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
>> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>
>> here is the link to matrix A and vector b:
>> https://drive.google.com/drive/folders/16YQPTK6TfXC6pV5RMdJ9g7X-ZiqbvwU8?usp=sharing
>>
>> I redid the problem (twice) by trying to solve a 1M finite elements
>> problem corresponding to ~ 4M n and 417M nnz matrix elements on the login
>> shell which has ~550GB mem, but it failed. First time it failed because of
>> bus error, second time it said killed. I have attached valgrind file from
>> both runs.
>>
>> OpenMPI is not my favorite. You need to use a suppressions file to get
>>> rid of all of that noise. Here is one:
>>>
>>
>> Thanks I have been using it, but sometimes I still see same amount of
>> errors.
>>
>>
>>
>> On Fri, Jan 18, 2019 at 3:12 AM Zhang, Junchao <jczhang at mcs.anl.gov>
>> wrote:
>>
>>> Usually when I meet a SEGV error, I will run it again with a parallel
>>> debugger like DDT and wait for it to segfault, and then examine the stack
>>> trace to see what is wrong.
>>> Can you compress your matrix and upload it to google drive, so we can
>>> try to reproduce the error.
>>> --Junchao Zhang
>>>
>>>
>>> On Thu, Jan 17, 2019 at 10:44 AM Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> I did two runs, one with the SuperLU_dist and one with bcgs using
>>>> jacobi, attached are the results of one of the reports from valgrind on one
>>>> random processor (out of the 128 files).
>>>>
>>>> DS = direct solver
>>>> IS = iterative  solver
>>>>
>>>> There is an awful lot of errors.
>>>>
>>>> how I initiated the two runs:
>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs
>>>> -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce
>>>> -ksp_monitor -log_view
>>>>
>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
>>>> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
>>>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>>>
>>>> Thank you
>>>>
>>>> On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>
>>>>>> 1) Running out of memory
>>>>>>>
>>>>>>> 2) You passed an invalid array
>>>>>>>
>>>>>> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script,
>>>>>> i.e. using 300GB/node, a total of 1200GB memory, using 4 nodes and 32
>>>>>> processors per node (128 processors in total).
>>>>>> I am not sure what would constitute an invalid array or how I can
>>>>>> check that. I am using the same procedure as when dealing with the smaller
>>>>>> matrix. i.e. Generate matrix A and vector b using FEM software then convert
>>>>>> the matrix and vector using a python script ready for petsc. read in petsc
>>>>>> and calculate.
>>>>>>
>>>>>> Are you running with 64-bit ints here?
>>>>>>>
>>>>>> Yes I have it configured petsc with  --with-64-bit-indices and
>>>>>> debugging mode, which this was run on.
>>>>>>
>>>>>
>>>>> It sounds like you have enough memory, but the fact that is runs for
>>>>> smaller problems makes me suspicious. It
>>>>> could still be a memory overwrite. Can you either
>>>>>
>>>>> a) Run under valgrind
>>>>>
>>>>> or
>>>>>
>>>>> b) Run under the debugger and get a stack trace
>>>>>
>>>>>  ?
>>>>>
>>>>>    Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>>>
>>>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>>>>
>>>>>>>> I will try that, although I think solving the real problem (later
>>>>>>>> on if I can get this to work) with 30 million finite elements might be a
>>>>>>>> problem for SuperLU_dist. so it is better to get an iterative solver to
>>>>>>>> work with first.
>>>>>>>>
>>>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>>>
>>>>>>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>>>>>>> -ksp_monitor -log_view
>>>>>>>> I have attached the full error output.
>>>>>>>>
>>>>>>>
>>>>>>> You are getting an SEGV on MatSetValues(), so its either
>>>>>>>
>>>>>>> 1) Running out of memory
>>>>>>>
>>>>>>> 2) You passed an invalid array
>>>>>>>
>>>>>>>
>>>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>>>>> None of our tests show this problem.
>>>>>>>>>
>>>>>>>> I am not how I can do that, but i have added my code which is quite
>>>>>>>> short and should only read and solve the system, the problem arises at
>>>>>>>> larger matrices for example current test case has 6 million finite elements
>>>>>>>> (~2B non-zero numbers and 25M x 25M matrix).
>>>>>>>>
>>>>>>>
>>>>>>> Are you running with 64-bit ints here?
>>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> The memory requested is an insane number. You may need to use 64
>>>>>>>>>>> bit integers.
>>>>>>>>>>
>>>>>>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the
>>>>>>>>>> process it says I can no longer use MUMPS and SuperLU as they are not
>>>>>>>>>> supported (I see on MUMPS webpage it supports 64int). However it does not
>>>>>>>>>> exactly solve the problem.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> This time, it crashes at
>>>>>>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ()
>>>>>>>>>>> line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>>>>>>> which allocates local portion of B^T*A.
>>>>>>>>>>> You may also try to increase number of cores to reduce local
>>>>>>>>>>> matrix size.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So I increased the number of cores to 16 on one node and ran it
>>>>>>>>>> by :
>>>>>>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>>>>>>> It crashed after reading in the matrix and before starting to
>>>>>>>>>> solve. The error:
>>>>>>>>>>
>>>>>>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [2]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>> [3]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [8]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [12]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01
>>>>>>>>>> exited on signal 9 (Killed).
>>>>>>>>>>
>>>>>>>>>> Now I was running this with valgrind as someone had previously
>>>>>>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Okay, its possible that there are bugs in the MPI implementation.
>>>>>>>>> So
>>>>>>>>>
>>>>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>>>
>>>>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>>>>> None of our tests show this problem.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> ==25940== Invalid read of size 8
>>>>>>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>>>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>>>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>>>> (mpits.c:373)
>>>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq
>>>>>>>>>> (mpits.c:572)
>>>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size
>>>>>>>>>> 16 alloc'd
>>>>>>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>>>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>>>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>>>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>>>> (mpits.c:371)
>>>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq
>>>>>>>>>> (mpits.c:572)
>>>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>>>> ==25940==
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Fande:
>>>>>>>>>>>
>>>>>>>>>>>> According to this PR
>>>>>>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>>>>>>
>>>>>>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>>>>>>
>>>>>>>>>>> Sure, we can. But I feel we need do more tests to compare
>>>>>>>>>>> scalable and non-scalable algorithms.
>>>>>>>>>>> On theory, for small to medium matrices, non-scalable
>>>>>>>>>>> matmatmult() algorithm enables more efficient
>>>>>>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>>>>>>> Hong
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via petsc-users
>>>>>>>>>>>>> <petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to allocating
>>>>>>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not
>>>>>>>>>>>>>> properly
>>>>>>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Probably the matrix is too large. You can try with more
>>>>>>>>>>>>>> compute nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a
>>>>>>>>>>>>>>> matrix of 25M x 25M I get the following error:
>>>>>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>>>>>>>>>>>>>>> Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack
>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>>>>>>>> are not available,
>>>>>>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start
>>>>>>>>>>>>>>> of the function
>>>>>>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line
>>>>>>>>>>>>>>> 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line
>>>>>>>>>>>>>>> 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>>>>>>>>>>>>>> trouble shooting.
>>>>>>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named
>>>>>>>>>>>>>>> r02g03 by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>>>>>>> [4]PETSC ERROR: Configure options
>>>>>>>>>>>>>>> PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>>>>>>>>> unknown file
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator
>>>>>>>>>>>>>>> MPI_COMM_WORLD
>>>>>>>>>>>>>>> with errorcode 59.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>>>>>>> processes.
>>>>>>>>>>>>>>> You may or may not see output from other processes,
>>>>>>>>>>>>>>> depending on
>>>>>>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Using Valgrind on only one of the valgrind files the
>>>>>>>>>>>>>>> following error was written:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays
>>>>>>>>>>>>>>> (aij.c:4445)
>>>>>>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult (matrix.c:9984)
>>>>>>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>>>>>>> (recently) free'd
>>>>>>>>>>>>>>> ==9053==
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <tempohoper at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I
>>>>>>>>>>>>>>>> ran the code again with the following options:
>>>>>>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>>>>>>> gamg -log_view
>>>>>>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>>>>>>> the log_view.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error
>>>>>>>>>>>>>>>>>> changes slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This likely implies that you have a memory error in your
>>>>>>>>>>>>>>>>> code (a memory leak would not cause this behaviour).
>>>>>>>>>>>>>>>>> I strongly suggest you make sure your code is free of
>>>>>>>>>>>>>>>>> memory errors.
>>>>>>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running it
>>>>>>>>>>>>>>>>>> with -malloc_dump, but not getting anything printed out, however, when run
>>>>>>>>>>>>>>>>>> with -log_view I see that Viewer is created 4 times, but destroyed 3 times.
>>>>>>>>>>>>>>>>>> The way I see it, I have destroyed it where I see I no longer have use for
>>>>>>>>>>>>>>>>>> it so not sure if I am wrong. Could this be the reason why it keeps
>>>>>>>>>>>>>>>>>> crashing? It crashes as soon as it reads the matrix, before entering the
>>>>>>>>>>>>>>>>>> solving mode (I have a print statement before solving starts that never
>>>>>>>>>>>>>>>>>> prints).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> how I run it in the job script on 2 node with 32
>>>>>>>>>>>>>>>>>> processors using the clusters OpenMPI.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/6b0a1245/attachment-0001.html>

From knepley at gmail.com  Tue Jan 29 08:16:14 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Jan 2019 09:16:14 -0500
Subject: [petsc-users] MPI Iterative solver crash on HPC
In-Reply-To: <CAE3k7C7AuKFpVM=SPM7HUNMFjmR7bKO_FBpkPL-b1_sSWmKUag@mail.gmail.com>
References: <CAE3k7C7NwnCnNyQ+CGoTEqoXu3GJV4b_7cGjhjRDcpLqdbg1kQ@mail.gmail.com>
	<CAJ98EDpP0J9E0dyTkDka4BqYyr6FvtjHZjhUSTjGA1Y31s2nNw@mail.gmail.com>
	<CAE3k7C606Fo2vostBGF1WaPqgr8n5zxWyMgfguViOtxpJaKvfg@mail.gmail.com>
	<CAE3k7C6Tgi05sR5m120MNWF1sj9m8Hyx-WD3xQ5Q5FJA8TAwEw@mail.gmail.com>
	<CA+MQGp8CAK6-o0wbXbsszkxiVF4fXG9yh1zQ0pyQNT_dKyWWvQ@mail.gmail.com>
	<CAGCphBu9qSz7G_TJvKiY1f1RAE98fPmQ40YgeZr=mxQwEMTfyQ@mail.gmail.com>
	<CAN5Wd-KkvFEvHx02GFLFUDnOuQW+YmnG_5De9Ln10oLgCB=bVg@mail.gmail.com>
	<CAGCphBsotsaJvT2=ScTioPPn51QV3epKZ8PEO9s3fn_ysaoYEg@mail.gmail.com>
	<CAE3k7C6y7zVgU3Zba_Od=bHtOkq-93Mwr3qWy-t0xg82WzXcXQ@mail.gmail.com>
	<CAMYG4GmMrgLr+VDaAeu+29G6dZn82YoRVzHMvfmzJMTkQQaexQ@mail.gmail.com>
	<CAE3k7C7TesUWuaq4QrT6oBQk3EkwgxhiT8AhH6xise6629OPrA@mail.gmail.com>
	<CAMYG4G=p=9_veoCjoSj84TN8OrS1RcO=KZGvZhr7JFEmJ1G-nA@mail.gmail.com>
	<CAE3k7C7xkKV-krYA5N_=gUYuVXEKH_HDwN+HaZ90knRp945ESg@mail.gmail.com>
	<CAMYG4GmM5rcV4v=mninfUMvtQY5GGYQhdLM56q-3u3wqAQzewQ@mail.gmail.com>
	<CAE3k7C6wxo40d79kuVGOMOHSET+NsUfr3M+-ZeV3zJf3vzDQgQ@mail.gmail.com>
	<CA+MQGp8WbFtpX1YOtgv6VqmDa=SvjCO7as2=MUNLSDUUzBByRQ@mail.gmail.com>
	<CAE3k7C4C4Ues6Ssu1SCPKYkPAdnMGE7n0itQDavbH1jKzTzCrA@mail.gmail.com>
	<CA+MQGp-_Ek=MAyr8P=T_-QVXSfN+ONjDrTdBdEeTBY-mLWLkAA@mail.gmail.com>
	<CAE3k7C7AuKFpVM=SPM7HUNMFjmR7bKO_FBpkPL-b1_sSWmKUag@mail.gmail.com>
Message-ID: <CAMYG4Gn3v7baAxgdONtBA3uV0KH+wtOjqPk-t7WyeF=UN7ncqg@mail.gmail.com>

On Tue, Jan 29, 2019 at 9:07 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thank you Zhang for that, I am a bit confused by the terminology maybe,
> but when you solved it on 16nodes (each node 128GB) with 576ranks that
> should give 2048GB (~2TB) of memory in total right? Does this mean the MPI
> does not work for SuperLU_dist?
>

No, He is telling you that SuperLU is running out of memory in parallel
(there is memory overhead beyond what it takes on
a single node, and this exhausts the extra). Direct solvers are not memory
scalable, and you have hit the limit.


> Unfortunately I cannot run the code on one node, as our nodes are limited
> to 500GB memory, hence I was hoping I could utilise several nodes and more
> memory.
>
> If anyone else has a solution for an iterative solver please do recommend
> as I am a bit stuck since so far and the combinations I have tried did not
> work.
>

The first step is to reduce your problem to the important linear algebraic
contributions, and then look in the literature
for a solver that worked in that case. Reproducing it in PETSc should
hopefully be easy.

 Thanks,

    Matt


>
> On Fri, Jan 25, 2019 at 5:32 PM Zhang, Junchao <jczhang at mcs.anl.gov>
> wrote:
>
>> Hi, Sal Am,
>>  I did some testes with your matrix and vector. It is a complex matrix
>> with N=4.7M and nnz=417M. Firstly, I tested on a machine with 36 cores and
>> 128GB memory on each compute node. I tried with direct solver and iterative
>> solver but both failed. For example, with 36 ranks on one compute node, I
>> got
>>
>> [9]PETSC ERROR: [9] SuperLU_DIST:pzgssvx line 465
>> /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>> [9]PETSC ERROR: [9] MatLUFactorNumeric_SuperLU_DIST line 314
>> /blues/gpfs/home/jczhang/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>>
>>   With 16 nodes, 576 ranks. I got
>>
>> SUPERLU_MALLOC fails for GAstore->rnzval[] at line 240 in file
>> /blues/gpfs/home/jczhang/petsc/bdw-dbg-complex/externalpackages/git.superlu_dist/SRC/pzutil.c
>>
>>   Next, I moved to another single-node machine with 1.5TB memory. It did
>> not fail this time. It ran overnight and is still doing superlu. Using the
>> top command, I found at peak, it consumed almost all memory. At stable
>> period, with 36 ranks, each rank consumed about 20GB memory. When I changed
>> to iterative solvers with -ksp_type bcgs -pc_type gamg
>> -mattransposematmult_via scalable, I did not meet errors seen on the
>> smaller memory machine. But the residual did not converge.
>>   So, I think the errors you met were simply out of memory error either
>> in superlu or in petsc.  If you have machines with large memory, you can
>> try it on. Otherwise, I let other petsc developers suggest better iterative
>> solvers to you.
>>   Thanks.
>>
>>
>> --Junchao Zhang
>>
>>
>> On Wed, Jan 23, 2019 at 2:52 AM Sal Am <tempohoper at gmail.com> wrote:
>>
>>> Sorry it took long had to see if I could shrink down the problem files
>>> from 50GB to something smaller (now ~10GB).
>>>
>>>> Can you compress your matrix and upload it to google drive, so we can
>>>> try to reproduce the error.
>>>>
>>>
>>> How I ran the problem: mpiexec valgrind --tool=memcheck
>>> --suppressions=$HOME/valgrind/valgrind-openmpi.supp -q --num-callers=20
>>> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
>>> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
>>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>>
>>> here is the link to matrix A and vector b:
>>> https://drive.google.com/drive/folders/16YQPTK6TfXC6pV5RMdJ9g7X-ZiqbvwU8?usp=sharing
>>>
>>> I redid the problem (twice) by trying to solve a 1M finite elements
>>> problem corresponding to ~ 4M n and 417M nnz matrix elements on the login
>>> shell which has ~550GB mem, but it failed. First time it failed because of
>>> bus error, second time it said killed. I have attached valgrind file from
>>> both runs.
>>>
>>> OpenMPI is not my favorite. You need to use a suppressions file to get
>>>> rid of all of that noise. Here is one:
>>>>
>>>
>>> Thanks I have been using it, but sometimes I still see same amount of
>>> errors.
>>>
>>>
>>>
>>> On Fri, Jan 18, 2019 at 3:12 AM Zhang, Junchao <jczhang at mcs.anl.gov>
>>> wrote:
>>>
>>>> Usually when I meet a SEGV error, I will run it again with a parallel
>>>> debugger like DDT and wait for it to segfault, and then examine the stack
>>>> trace to see what is wrong.
>>>> Can you compress your matrix and upload it to google drive, so we can
>>>> try to reproduce the error.
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Thu, Jan 17, 2019 at 10:44 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> I did two runs, one with the SuperLU_dist and one with bcgs using
>>>>> jacobi, attached are the results of one of the reports from valgrind on one
>>>>> random processor (out of the 128 files).
>>>>>
>>>>> DS = direct solver
>>>>> IS = iterative  solver
>>>>>
>>>>> There is an awful lot of errors.
>>>>>
>>>>> how I initiated the two runs:
>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>> --log-file=valgrind.log-IS.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>> -pc_type jacobi -mattransposematmult_via scalable -build_twosided allreduce
>>>>> -ksp_monitor -log_view
>>>>>
>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>> --log-file=valgrind.log-DS.%p ./solveCSys -malloc off -ksp_type gmres
>>>>> -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_max_it 1
>>>>> -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged
>>>>>
>>>>> Thank you
>>>>>
>>>>> On Thu, Jan 17, 2019 at 4:24 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Jan 17, 2019 at 9:18 AM Sal Am <tempohoper at gmail.com> wrote:
>>>>>>
>>>>>>> 1) Running out of memory
>>>>>>>>
>>>>>>>> 2) You passed an invalid array
>>>>>>>>
>>>>>>> I have select=4:ncpus=32:mpiprocs=32:mem=300GB in the job script,
>>>>>>> i.e. using 300GB/node, a total of 1200GB memory, using 4 nodes and 32
>>>>>>> processors per node (128 processors in total).
>>>>>>> I am not sure what would constitute an invalid array or how I can
>>>>>>> check that. I am using the same procedure as when dealing with the smaller
>>>>>>> matrix. i.e. Generate matrix A and vector b using FEM software then convert
>>>>>>> the matrix and vector using a python script ready for petsc. read in petsc
>>>>>>> and calculate.
>>>>>>>
>>>>>>> Are you running with 64-bit ints here?
>>>>>>>>
>>>>>>> Yes I have it configured petsc with  --with-64-bit-indices and
>>>>>>> debugging mode, which this was run on.
>>>>>>>
>>>>>>
>>>>>> It sounds like you have enough memory, but the fact that is runs for
>>>>>> smaller problems makes me suspicious. It
>>>>>> could still be a memory overwrite. Can you either
>>>>>>
>>>>>> a) Run under valgrind
>>>>>>
>>>>>> or
>>>>>>
>>>>>> b) Run under the debugger and get a stack trace
>>>>>>
>>>>>>  ?
>>>>>>
>>>>>>    Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> On Thu, Jan 17, 2019 at 1:59 PM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Jan 17, 2019 at 8:16 AM Sal Am <tempohoper at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in parallel?
>>>>>>>>>>
>>>>>>>>> I will try that, although I think solving the real problem (later
>>>>>>>>> on if I can get this to work) with 30 million finite elements might be a
>>>>>>>>> problem for SuperLU_dist. so it is better to get an iterative solver to
>>>>>>>>> work with first.
>>>>>>>>>
>>>>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>>>>
>>>>>>>>> How I ran it: mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>> --log-file=valgrind.log-osaii.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>>>> -pc_type gamg -mattransposematmult_via scalable -build_twosided allreduce
>>>>>>>>> -ksp_monitor -log_view
>>>>>>>>> I have attached the full error output.
>>>>>>>>>
>>>>>>>>
>>>>>>>> You are getting an SEGV on MatSetValues(), so its either
>>>>>>>>
>>>>>>>> 1) Running out of memory
>>>>>>>>
>>>>>>>> 2) You passed an invalid array
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2) Is it possible to get something that fails here but we can run.
>>>>>>>>>> None of our tests show this problem.
>>>>>>>>>>
>>>>>>>>> I am not how I can do that, but i have added my code which is
>>>>>>>>> quite short and should only read and solve the system, the problem arises
>>>>>>>>> at larger matrices for example current test case has 6 million finite
>>>>>>>>> elements (~2B non-zero numbers and 25M x 25M matrix).
>>>>>>>>>
>>>>>>>>
>>>>>>>> Are you running with 64-bit ints here?
>>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Jan 16, 2019 at 1:12 PM Matthew Knepley <knepley at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Jan 16, 2019 at 3:52 AM Sal Am via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> The memory requested is an insane number. You may need to use 64
>>>>>>>>>>>> bit integers.
>>>>>>>>>>>
>>>>>>>>>>> Thanks Mark, I reconfigured it to use 64bit, however in the
>>>>>>>>>>> process it says I can no longer use MUMPS and SuperLU as they are not
>>>>>>>>>>> supported (I see on MUMPS webpage it supports 64int). However it does not
>>>>>>>>>>> exactly solve the problem.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SuperLU_dist supports 64-bit ints. Are you not running in
>>>>>>>>>> parallel?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> This time, it crashes at
>>>>>>>>>>>> [6]PETSC ERROR: #1 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ()
>>>>>>>>>>>> line 1989 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>> ierr      = PetscMalloc1(bi[pn]+1,&bj);
>>>>>>>>>>>> which allocates local portion of B^T*A.
>>>>>>>>>>>> You may also try to increase number of cores to reduce local
>>>>>>>>>>>> matrix size.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So I increased the number of cores to 16 on one node and ran it
>>>>>>>>>>> by :
>>>>>>>>>>> mpiexec valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>> --log-file=valgrind.log-osa.%p ./solveCSys -malloc off -ksp_type bcgs
>>>>>>>>>>> -pc_type gamg -mattransposematmult_via scalable -ksp_monitor -log_view
>>>>>>>>>>> It crashed after reading in the matrix and before starting to
>>>>>>>>>>> solve. The error:
>>>>>>>>>>>
>>>>>>>>>>> [15]PETSC ERROR: [0]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [2]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>> [3]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [4]PETSC ERROR: [5]PETSC ERROR: [6]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [8]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [12]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [12]PETSC ERROR: [14]PETSC ERROR:
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process
>>>>>>>>>>> (or the batch system) has told this process to end
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>> mpiexec noticed that process rank 10 with PID 0 on node r03n01
>>>>>>>>>>> exited on signal 9 (Killed).
>>>>>>>>>>>
>>>>>>>>>>> Now I was running this with valgrind as someone had previously
>>>>>>>>>>> suggested and the 16 files created all contain the same type of error:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Okay, its possible that there are bugs in the MPI implementation.
>>>>>>>>>> So
>>>>>>>>>>
>>>>>>>>>> 1) Try using    -build_twosided allreduce    on this run
>>>>>>>>>>
>>>>>>>>>> 2) Is it possible to get something that fails here but we can
>>>>>>>>>> run. None of our tests show this problem.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> ==25940== Invalid read of size 8
>>>>>>>>>>> ==25940==    at 0x5103326: PetscCheckPointer (checkptr.c:81)
>>>>>>>>>>> ==25940==    by 0x4F42058: PetscCommGetNewTag (tagm.c:77)
>>>>>>>>>>> ==25940==    by 0x4FC952D: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>>>>> (mpits.c:373)
>>>>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq
>>>>>>>>>>> (mpits.c:572)
>>>>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>>>>> ==25940==  Address 0x19f807fc is 12 bytes inside a block of size
>>>>>>>>>>> 16 alloc'd
>>>>>>>>>>> ==25940==    at 0x4C2A603: memalign (vg_replace_malloc.c:899)
>>>>>>>>>>> ==25940==    by 0x4FD0B0E: PetscMallocAlign (mal.c:41)
>>>>>>>>>>> ==25940==    by 0x4FD23E7: PetscMallocA (mal.c:397)
>>>>>>>>>>> ==25940==    by 0x4FC948E: PetscCommBuildTwoSidedFReq_Ibarrier
>>>>>>>>>>> (mpits.c:371)
>>>>>>>>>>> ==25940==    by 0x4FCB29B: PetscCommBuildTwoSidedFReq
>>>>>>>>>>> (mpits.c:572)
>>>>>>>>>>> ==25940==    by 0x52BBFF4: VecAssemblyBegin_MPI_BTS (pbvec.c:251)
>>>>>>>>>>> ==25940==    by 0x52D6B42: VecAssemblyBegin (vector.c:140)
>>>>>>>>>>> ==25940==    by 0x5328C97: VecLoad_Binary (vecio.c:141)
>>>>>>>>>>> ==25940==    by 0x5329051: VecLoad_Default (vecio.c:516)
>>>>>>>>>>> ==25940==    by 0x52E0BAB: VecLoad (vector.c:933)
>>>>>>>>>>> ==25940==    by 0x4013D5: main (solveCmplxLinearSys.cpp:31)
>>>>>>>>>>> ==25940==
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 14, 2019 at 7:29 PM Zhang, Hong <hzhang at mcs.anl.gov>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Fande:
>>>>>>>>>>>>
>>>>>>>>>>>>> According to this PR
>>>>>>>>>>>>> https://bitbucket.org/petsc/petsc/pull-requests/1061/a_selinger-feature-faster-scalable/diff
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should we set the scalable algorithm as default?
>>>>>>>>>>>>>
>>>>>>>>>>>> Sure, we can. But I feel we need do more tests to compare
>>>>>>>>>>>> scalable and non-scalable algorithms.
>>>>>>>>>>>> On theory, for small to medium matrices, non-scalable
>>>>>>>>>>>> matmatmult() algorithm enables more efficient
>>>>>>>>>>>> data accessing. Andreas optimized scalable implementation. Our
>>>>>>>>>>>> non-scalable implementation might have room to be further optimized.
>>>>>>>>>>>> Hong
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 10:34 AM Zhang, Hong via petsc-users <
>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Add option '-mattransposematmult_via scalable'
>>>>>>>>>>>>>> Hong
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 9:52 AM Zhang, Junchao via
>>>>>>>>>>>>>> petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I saw the following error message in your first email.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [0]PETSC ERROR: Out of memory. This could be due to
>>>>>>>>>>>>>>> allocating
>>>>>>>>>>>>>>> [0]PETSC ERROR: too large an object or bleeding by not
>>>>>>>>>>>>>>> properly
>>>>>>>>>>>>>>> [0]PETSC ERROR: destroying unneeded objects.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Probably the matrix is too large. You can try with more
>>>>>>>>>>>>>>> compute nodes, for example, use 8 nodes instead of 2, and see what happens.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 7:45 AM Sal Am via petsc-users <
>>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Using a larger problem set with 2B non-zero elements and a
>>>>>>>>>>>>>>>> matrix of 25M x 25M I get the following error:
>>>>>>>>>>>>>>>> [4]PETSC ERROR:
>>>>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>>>>>>>>>>>>>>> Violation, probably memory access out of range
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>>>>> [4]PETSC ERROR: or see
>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux
>>>>>>>>>>>>>>>> and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>> [4]PETSC ERROR: likely location of problem given in stack
>>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>>> [4]PETSC ERROR: ---------------------  Stack Frames
>>>>>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack
>>>>>>>>>>>>>>>> are not available,
>>>>>>>>>>>>>>>> [4]PETSC ERROR:       INSTEAD the line number of the start
>>>>>>>>>>>>>>>> of the function
>>>>>>>>>>>>>>>> [4]PETSC ERROR:       is given.
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatCreateSeqAIJWithArrays line 4422
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/seq/aij.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatMatMultSymbolic_SeqAIJ_SeqAIJ line
>>>>>>>>>>>>>>>> 747 /lustre/home/vef002/petsc/src/mat/impls/aij/seq/matmatmult.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4]
>>>>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable line 1256
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line
>>>>>>>>>>>>>>>> 1156 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] MatTransposeMatMult line 9950
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/mat/interface/matrix.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCGAMGCoarsen_AGG line 871
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp_GAMG line 428
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] PCSetUp line 894
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: [4] KSPSetUp line 304
>>>>>>>>>>>>>>>> /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>>>>>>>>>>>> [4]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>> [4]PETSC ERROR: See
>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>>>>>>>>>>>>>>> trouble shooting.
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
>>>>>>>>>>>>>>>> [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named
>>>>>>>>>>>>>>>> r02g03 by vef002 Fri Jan 11 09:13:23 2019
>>>>>>>>>>>>>>>> [4]PETSC ERROR: Configure options
>>>>>>>>>>>>>>>> PETSC_ARCH=linux-cumulus-debug
>>>>>>>>>>>>>>>> --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc
>>>>>>>>>>>>>>>> --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort
>>>>>>>>>>>>>>>> --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx
>>>>>>>>>>>>>>>> --download-parmetis --download-metis --download-ptscotch
>>>>>>>>>>>>>>>> --download-superlu_dist --download-mumps --with-scalar-type=complex
>>>>>>>>>>>>>>>> --with-debugging=yes --download-scalapack --download-superlu
>>>>>>>>>>>>>>>> --download-fblaslapack=1 --download-cmake
>>>>>>>>>>>>>>>> [4]PETSC ERROR: #1 User provided function() line 0 in
>>>>>>>>>>>>>>>> unknown file
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>> MPI_ABORT was invoked on rank 4 in communicator
>>>>>>>>>>>>>>>> MPI_COMM_WORLD
>>>>>>>>>>>>>>>> with errorcode 59.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>>>>>>>>>>>>>> processes.
>>>>>>>>>>>>>>>> You may or may not see output from other processes,
>>>>>>>>>>>>>>>> depending on
>>>>>>>>>>>>>>>> exactly when Open MPI kills them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR:
>>>>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some
>>>>>>>>>>>>>>>> process (or the batch system) has told this process to end
>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>>>>>>>>>>>>>>> -on_error_attach_debugger
>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see
>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Using Valgrind on only one of the valgrind files the
>>>>>>>>>>>>>>>> following error was written:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ==9053== Invalid read of size 4
>>>>>>>>>>>>>>>> ==9053==    at 0x5B8067E: MatCreateSeqAIJWithArrays
>>>>>>>>>>>>>>>> (aij.c:4445)
>>>>>>>>>>>>>>>> ==9053==    by 0x5BC2608: MatMatMultSymbolic_SeqAIJ_SeqAIJ
>>>>>>>>>>>>>>>> (matmatmult.c:790)
>>>>>>>>>>>>>>>> ==9053==    by 0x5D106F8:
>>>>>>>>>>>>>>>> MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ_nonscalable (mpimatmatmult.c:1337)
>>>>>>>>>>>>>>>> ==9053==    by 0x5D0E84E: MatTransposeMatMult_MPIAIJ_MPIAIJ
>>>>>>>>>>>>>>>> (mpimatmatmult.c:1186)
>>>>>>>>>>>>>>>> ==9053==    by 0x5457C57: MatTransposeMatMult
>>>>>>>>>>>>>>>> (matrix.c:9984)
>>>>>>>>>>>>>>>> ==9053==    by 0x64DD99D: PCGAMGCoarsen_AGG (agg.c:882)
>>>>>>>>>>>>>>>> ==9053==    by 0x64C7527: PCSetUp_GAMG (gamg.c:522)
>>>>>>>>>>>>>>>> ==9053==    by 0x6592AA0: PCSetUp (precon.c:932)
>>>>>>>>>>>>>>>> ==9053==    by 0x66B1267: KSPSetUp (itfunc.c:391)
>>>>>>>>>>>>>>>> ==9053==    by 0x4019A2: main (solveCmplxLinearSys.cpp:68)
>>>>>>>>>>>>>>>> ==9053==  Address 0x8386997f4 is not stack'd, malloc'd or
>>>>>>>>>>>>>>>> (recently) free'd
>>>>>>>>>>>>>>>> ==9053==
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jan 11, 2019 at 8:41 AM Sal Am <
>>>>>>>>>>>>>>>> tempohoper at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you Dave,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I reconfigured PETSc with valgrind and debugging mode, I
>>>>>>>>>>>>>>>>> ran the code again with the following options:
>>>>>>>>>>>>>>>>> mpiexec -n 8 valgrind --tool=memcheck -q --num-callers=20
>>>>>>>>>>>>>>>>> --log-file=valgrind.log.%p ./solveCSys -malloc off -ksp_type bcgs -pc_type
>>>>>>>>>>>>>>>>> gamg -log_view
>>>>>>>>>>>>>>>>> (as on the petsc website you linked)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It finished solving using the iterative solver, but the
>>>>>>>>>>>>>>>>> resulting valgrind.log.%p files (all 8 corresponding to each processor) are
>>>>>>>>>>>>>>>>> all empty. And it took a whooping ~15hours, for what used to take
>>>>>>>>>>>>>>>>> ~10-20min. Maybe this is because of valgrind? I am not sure. Attached is
>>>>>>>>>>>>>>>>> the log_view.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 10, 2019 at 8:59 AM Dave May <
>>>>>>>>>>>>>>>>> dave.mayhem23 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, 10 Jan 2019 at 08:55, Sal Am via petsc-users <
>>>>>>>>>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am not sure what is exactly is wrong as the error
>>>>>>>>>>>>>>>>>>> changes slightly every time I run it (without changing the parameters).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This likely implies that you have a memory error in your
>>>>>>>>>>>>>>>>>> code (a memory leak would not cause this behaviour).
>>>>>>>>>>>>>>>>>> I strongly suggest you make sure your code is free of
>>>>>>>>>>>>>>>>>> memory errors.
>>>>>>>>>>>>>>>>>> You can do this using valgrind. See here
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for an explanation of how to use valgrind.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have attached the first two run's errors and my code.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there a memory leak somewhere? I have tried running
>>>>>>>>>>>>>>>>>>> it with -malloc_dump, but not getting anything printed out, however, when
>>>>>>>>>>>>>>>>>>> run with -log_view I see that Viewer is created 4 times, but destroyed 3
>>>>>>>>>>>>>>>>>>> times. The way I see it, I have destroyed it where I see I no longer have
>>>>>>>>>>>>>>>>>>> use for it so not sure if I am wrong. Could this be the reason why it keeps
>>>>>>>>>>>>>>>>>>> crashing? It crashes as soon as it reads the matrix, before entering the
>>>>>>>>>>>>>>>>>>> solving mode (I have a print statement before solving starts that never
>>>>>>>>>>>>>>>>>>> prints).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> how I run it in the job script on 2 node with 32
>>>>>>>>>>>>>>>>>>> processors using the clusters OpenMPI.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> mpiexec ./solveCSys -ksp_type bcgs -pc_type gamg
>>>>>>>>>>>>>>>>>>> -ksp_converged_reason -ksp_monitor_true_residual -log_view
>>>>>>>>>>>>>>>>>>> -ksp_error_if_not_converged -ksp_monitor -malloc_log -ksp_view
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the matrix:
>>>>>>>>>>>>>>>>>>> 2 122 821 366 (non-zero elements)
>>>>>>>>>>>>>>>>>>> 25 947 279 x 25 947 279
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks and all the best
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/cf8bc999/attachment-0001.html>

From chen2018 at purdue.edu  Tue Jan 29 11:39:51 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Tue, 29 Jan 2019 17:39:51 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
Message-ID: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>

Hello,


I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?


Thanks


block
    PetscErrorCode ierr
     PetscMPIInt rank,nproc, mystart
     PetscInt  nelem
     integer,allocatable  ::info(:)
     real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
     integer, allocatable      :: idx(:)
     PetscScalar, pointer :: xx_v(:)

     PC               prec
     PetscScalar ::   val
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
     PetscViewer viewer
     PetscInt  geq,j,k,ne,M,Istart,Iend
     PetscBool  flg
     KSPConvergedReason reason
     Vec         dummyVec, dummyVecs(1)
     MatNullSpace nullspace

     call PetscInitialize( PETSC_NULL_CHARACTER, ierr )

     if (ierr .ne. 0) then
     print*,'Unable to initialize PETSc'
     stop
     endif
     call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
     call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
     mystart=rank+1

 !   Parameter set
     info=ptsystem%getInitInfo()
     nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
     print*,'nelem',nelem
 !-------------------------------------
 !  Create Matrix
     call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
     call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
     call MatSetType( Amat, MATMPIBAIJ, ierr )
     ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
     call MatSetFromOptions( Amat, ierr )
     call MatSetUp( Amat, ierr )
     call MatGetOwnershipRange( Amat, Istart, Iend, ierr )

      xvec = tVec(0)
      call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
      call VecSetFromOptions( xvec, ierr )
      call VecDuplicate( xvec, bvec, ierr )
      call VecDuplicate( xvec, uvec, ierr )

   t1 = MPI_WTime();

      do i=mystart,nelem,nproc
            call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
            ne=size(idx)
            if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
             call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
      end do

      nelem   = info(Info_EleRHSNum)
      mystart = rank+1

      do i = mystart, nelem, nproc
        call ptSystem%getElementalRHS(i, RHSe, idx)
        print*,'idx',idx
        ne=size(idx)
        if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
      end do
      call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
      call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
      call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
     t2 = MPI_WTime();
      print*,'assembeling time',t2-t1
      ! Solve
      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.
      call KSPSetOperators(ksp,Amat,Amat,ierr)
      call KSPSetFromOptions(ksp,ierr)
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    call KSPSetType(ksp,KSPCG,ierr)
    call KSPGetPC(ksp,prec,ierr)
   ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
    call PCSetType(prec,PCJACOBI,ierr)
     call KSPSolve(ksp,bvec,xvec,ierr)
     call PetscFinalize(ierr)

end block



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/92ad96cf/attachment.html>

From song.gao2 at mail.mcgill.ca  Tue Jan 29 12:22:43 2019
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Tue, 29 Jan 2019 13:22:43 -0500
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>

I think you would prefer to preallocate the matrix

uncomment this line
! call
MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
ierr)



Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <
petsc-users at mcs.anl.gov> a ?crit :

> Hello,
>
>
> I have a 2D system which is assembled by each elemental matrix.  Ae is my
> elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side,
> idx is the global index. My code is as follow, however ,the assembling rate
> is super slow(Marked red in the code). I am not sure whether the assembling
> type is right or not. Since for each element, idx are not continuous
> numbers. Do you have any idea what is the better way to assemble the matrix?
>
>
> Thanks
>
>
> block
>     PetscErrorCode ierr
>      PetscMPIInt rank,nproc, mystart
>      PetscInt  nelem
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
>      integer, allocatable      :: idx(:)
>      PetscScalar, pointer :: xx_v(:)
>
>      PC               prec
>      PetscScalar ::   val
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>      PetscViewer viewer
>      PetscInt  geq,j,k,ne,M,Istart,Iend
>      PetscBool  flg
>      KSPConvergedReason reason
>      Vec         dummyVec, dummyVecs(1)
>      MatNullSpace nullspace
>
>      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
>
>      if (ierr .ne. 0) then
>      print*,'Unable to initialize PETSc'
>      stop
>      endif
>      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
>      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>      mystart=rank+1
>
>  !   Parameter set
>      info=ptsystem%getInitInfo()
>      nelem =
> info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
>      print*,'nelem',nelem
>  !-------------------------------------
>  !  Create Matrix
>      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
>      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1),
> ierr )
>      call MatSetType( Amat, MATMPIBAIJ, ierr )
>      ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>      call MatSetFromOptions( Amat, ierr )
>      call MatSetUp( Amat, ierr )
>      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
>
>       xvec = tVec(0)
>       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
>       call VecSetFromOptions( xvec, ierr )
>       call VecDuplicate( xvec, bvec, ierr )
>       call VecDuplicate( xvec, uvec, ierr )
>
>    t1 = MPI_WTime();
>
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             if (allocated(auxRHSe))  call
> VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
>              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
>       end do
>
>       nelem   = info(Info_EleRHSNum)
>       mystart = rank+1
>
>       do i = mystart, nelem, nproc
>         call ptSystem%getElementalRHS(i, RHSe, idx)
>         print*,'idx',idx
>         ne=size(idx)
>         if (allocated(RHSe))  call
> VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
>       end do
>       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       !  this part is slow, the for loop above is done but here it may get
> stuck
>       call VecAssemblyBegin(bvec,ierr)
>                                 ! For a 2500 DOF system, assembling only
> takes over 2 seconds
>       call VecAssemblyEnd(bvec,ierr)
>                                 ! But for a 10000 DOF system , it gets stuck
>      t2 = MPI_WTime();
>       print*,'assembeling time',t2-t1
>       ! Solve
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>       call KSPSetOperators(ksp,Amat,Amat,ierr)
>       call KSPSetFromOptions(ksp,ierr)
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     call KSPSetType(ksp,KSPCG,ierr)
>     call KSPGetPC(ksp,prec,ierr)
>    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
>     call PCSetType(prec,PCJACOBI,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>      call PetscFinalize(ierr)
>
> end block
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/b6f9cfb4/attachment.html>

From nahmad16 at ku.edu.tr  Tue Jan 29 13:36:40 2019
From: nahmad16 at ku.edu.tr (Najeeb Ahmad)
Date: Tue, 29 Jan 2019 22:36:40 +0300
Subject: [petsc-users] PETSc GPU
Message-ID: <CAJJ73m11jF_gYHrNfHrH9S1SwrrmqRMCfms9Gu1AKV=JTC-9QA@mail.gmail.com>

Hi all,

I am interested in knowing the state of the art in PETSc GPU
implementation. In the paper "Preliminary Implementation of PETSc Using
GPUs", it is mentioned that GPU implementation uses cusp and thrust
libraries. Is this currently true as well or does PETSc now use cuSPARSE or
any other libraries and/or any other improvements?

Best Regards,

-- 
*Najeeb Ahmad*


*Research and Teaching Assistant*
*PARallel and MultiCORE Computing Laboratory (ParCoreLab) *

*Computer Science and Engineering*
*Ko? University, Istanbul, Turkey*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/6a2c60a3/attachment.html>

From chen2018 at purdue.edu  Tue Jan 29 17:20:52 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Tue, 29 Jan 2019 23:20:52 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
Message-ID: <DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>

Hi Song,

I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(Mat<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat> B,PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> d_nz,const PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> d_nnz[],PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> o_nz,const PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> o_nnz[])the same as my elemental matrix?

[cid:ac5af973-49cd-4207-8f47-a94751d7f444]

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Song Gao <song.gao2 at mail.mcgill.ca>
Sent: Tuesday, January 29, 2019 1:22 PM
To: Yaxiong Chen
Cc: Matthew Knepley; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] PETSC matrix assembling super slow

I think you would prefer to preallocate the matrix

uncomment this line
! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)



Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> a ?crit :

Hello,


I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?


Thanks


block
    PetscErrorCode ierr
     PetscMPIInt rank,nproc, mystart
     PetscInt  nelem
     integer,allocatable  ::info(:)
     real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
     integer, allocatable      :: idx(:)
     PetscScalar, pointer :: xx_v(:)

     PC               prec
     PetscScalar ::   val
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
     PetscViewer viewer
     PetscInt  geq,j,k,ne,M,Istart,Iend
     PetscBool  flg
     KSPConvergedReason reason
     Vec         dummyVec, dummyVecs(1)
     MatNullSpace nullspace

     call PetscInitialize( PETSC_NULL_CHARACTER, ierr )

     if (ierr .ne. 0) then
     print*,'Unable to initialize PETSc'
     stop
     endif
     call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
     call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
     mystart=rank+1

 !   Parameter set
     info=ptsystem%getInitInfo()
     nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
     print*,'nelem',nelem
 !-------------------------------------
 !  Create Matrix
     call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
     call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
     call MatSetType( Amat, MATMPIBAIJ, ierr )
     ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
     call MatSetFromOptions( Amat, ierr )
     call MatSetUp( Amat, ierr )
     call MatGetOwnershipRange( Amat, Istart, Iend, ierr )

      xvec = tVec(0)
      call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
      call VecSetFromOptions( xvec, ierr )
      call VecDuplicate( xvec, bvec, ierr )
      call VecDuplicate( xvec, uvec, ierr )

   t1 = MPI_WTime();

      do i=mystart,nelem,nproc
            call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
            ne=size(idx)
            if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
             call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
      end do

      nelem   = info(Info_EleRHSNum)
      mystart = rank+1

      do i = mystart, nelem, nproc
        call ptSystem%getElementalRHS(i, RHSe, idx)
        print*,'idx',idx
        ne=size(idx)
        if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
      end do
      call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
      call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
      call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
     t2 = MPI_WTime();
      print*,'assembeling time',t2-t1
      ! Solve
      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.
      call KSPSetOperators(ksp,Amat,Amat,ierr)
      call KSPSetFromOptions(ksp,ierr)
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    call KSPSetType(ksp,KSPCG,ierr)
    call KSPGetPC(ksp,prec,ierr)
   ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
    call PCSetType(prec,PCJACOBI,ierr)
     call KSPSolve(ksp,bvec,xvec,ierr)
     call PetscFinalize(ierr)

end block



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/0f9fa5c2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3840 bytes
Desc: pastedImage.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/0f9fa5c2/attachment-0001.png>

From davelee2804 at gmail.com  Tue Jan 29 18:13:45 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Wed, 30 Jan 2019 11:13:45 +1100
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
Message-ID: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>

Hi,

just wondering if its possible to manually specify the parallel
decomposition of data within PETSc Vecs and Mats? And if so, will the
FieldSplit preconditioning handle this also?

I have an application (non-PETSc) data layout as:
DATA[16][4][num_procs][3472]

and if possible I would like to use FieldSplit preconditioning without
re-arranging this layout.

Currently I have a nested FieldSplit preconditioning with 16 outer field
splits, and a schur complement field split within each. however this throws
up an indexing error as FieldSplit assumes that the parallel decomposition
is contiguous (I think).

[0]PETSC ERROR: #1 ISComplement() line 750 in
/home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c

[0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
/home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c

[0]PETSC ERROR: #3 PCSetUp() line 923 in
/home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c

Cheers, Dave.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/91ad292b/attachment.html>

From mfadams at lbl.gov  Tue Jan 29 19:25:12 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 29 Jan 2019 20:25:12 -0500
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>

Slow assembly is often from not preallocating correctly. I am guessing that
you are using Q1 element and f9==9, and thus the preallocation should be OK
if this is a scalar problem on a regular grid and f6-==6 should be OK for
the off processor allocation, if my assumptions are correct.

You can run with -info, which will tell you how many allocation were done
in assembly. Make sure that it is small (eg, 0).

I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy
things with seeming simple code. You could just do this manually if you can
find anything else.

And I trust you are running an optimized version of PETSc.


On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi Song,
>
> I don't quite understand how I can use this command. I don't partition the
> gloabl matrix. If I add my elemental matrix to the global system it will be
> like this. And in my parallel part, I use each core to generate the
> elemental matrix in turn. In this case, I guess each core will be assigned
> the space for global matrix and finally be assembled.But according to the
> manual, it seems each core will store a part of the global matrix. Is the local
> submatrix in the MatMPIAIJSetPreallocation
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
> (Mat
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat>
> B,PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nnz[],PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nnz[])the same as my elemental matrix?
>
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Song Gao <song.gao2 at mail.mcgill.ca>
> *Sent:* Tuesday, January 29, 2019 1:22 PM
> *To:* Yaxiong Chen
> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> I think you would prefer to preallocate the matrix
>
> uncomment this line
> ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>
>
>
> Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <
> petsc-users at mcs.anl.gov> a ?crit :
>
> Hello,
>
>
> I have a 2D system which is assembled by each elemental matrix.  Ae is my
> elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side,
> idx is the global index. My code is as follow, however ,the assembling rate
> is super slow(Marked red in the code). I am not sure whether the assembling
> type is right or not. Since for each element, idx are not continuous
> numbers. Do you have any idea what is the better way to assemble the matrix?
>
>
> Thanks
>
>
> block
>     PetscErrorCode ierr
>      PetscMPIInt rank,nproc, mystart
>      PetscInt  nelem
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
>      integer, allocatable      :: idx(:)
>      PetscScalar, pointer :: xx_v(:)
>
>      PC               prec
>      PetscScalar ::   val
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>      PetscViewer viewer
>      PetscInt  geq,j,k,ne,M,Istart,Iend
>      PetscBool  flg
>      KSPConvergedReason reason
>      Vec         dummyVec, dummyVecs(1)
>      MatNullSpace nullspace
>
>      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
>
>      if (ierr .ne. 0) then
>      print*,'Unable to initialize PETSc'
>      stop
>      endif
>      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
>      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>      mystart=rank+1
>
>  !   Parameter set
>      info=ptsystem%getInitInfo()
>      nelem =
> info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
>      print*,'nelem',nelem
>  !-------------------------------------
>  !  Create Matrix
>      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
>      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1),
> ierr )
>      call MatSetType( Amat, MATMPIBAIJ, ierr )
>      ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>      call MatSetFromOptions( Amat, ierr )
>      call MatSetUp( Amat, ierr )
>      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
>
>       xvec = tVec(0)
>       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
>       call VecSetFromOptions( xvec, ierr )
>       call VecDuplicate( xvec, bvec, ierr )
>       call VecDuplicate( xvec, uvec, ierr )
>
>    t1 = MPI_WTime();
>
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             if (allocated(auxRHSe))  call
> VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
>              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
>       end do
>
>       nelem   = info(Info_EleRHSNum)
>       mystart = rank+1
>
>       do i = mystart, nelem, nproc
>         call ptSystem%getElementalRHS(i, RHSe, idx)
>         print*,'idx',idx
>         ne=size(idx)
>         if (allocated(RHSe))  call
> VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
>       end do
>       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       !  this part is slow, the for loop above is done but here it may get
> stuck
>       call VecAssemblyBegin(bvec,ierr)
>                                 ! For a 2500 DOF system, assembling only
> takes over 2 seconds
>       call VecAssemblyEnd(bvec,ierr)
>                                 ! But for a 10000 DOF system , it gets stuck
>      t2 = MPI_WTime();
>       print*,'assembeling time',t2-t1
>       ! Solve
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>       call KSPSetOperators(ksp,Amat,Amat,ierr)
>       call KSPSetFromOptions(ksp,ierr)
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     call KSPSetType(ksp,KSPCG,ierr)
>     call KSPGetPC(ksp,prec,ierr)
>    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
>     call PCSetType(prec,PCJACOBI,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>      call PetscFinalize(ierr)
>
> end block
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/d03ae250/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3840 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/d03ae250/attachment-0001.png>

From mfadams at lbl.gov  Tue Jan 29 19:34:59 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 29 Jan 2019 20:34:59 -0500
Subject: [petsc-users] PETSc GPU
In-Reply-To: <CAJJ73m11jF_gYHrNfHrH9S1SwrrmqRMCfms9Gu1AKV=JTC-9QA@mail.gmail.com>
References: <CAJJ73m11jF_gYHrNfHrH9S1SwrrmqRMCfms9Gu1AKV=JTC-9QA@mail.gmail.com>
Message-ID: <CADOhEh5eH+dnmWRSb-_8-sL-9CRORnPcyPFGMsWA90qCAQ2=AA@mail.gmail.com>

https://www.mcs.anl.gov/petsc/features/gpus.html

On Tue, Jan 29, 2019 at 2:38 PM Najeeb Ahmad via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi all,
>
> I am interested in knowing the state of the art in PETSc GPU
> implementation. In the paper "Preliminary Implementation of PETSc Using
> GPUs", it is mentioned that GPU implementation uses cusp and thrust
> libraries. Is this currently true as well or does PETSc now use cuSPARSE or
> any other libraries and/or any other improvements?
>
> Best Regards,
>
> --
> *Najeeb Ahmad*
>
>
> *Research and Teaching Assistant*
> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>
> *Computer Science and Engineering*
> *Ko? University, Istanbul, Turkey*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/84ce534b/attachment.html>

From mfadams at lbl.gov  Tue Jan 29 21:02:45 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 29 Jan 2019 22:02:45 -0500
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>

Optimized is a configuration flag not a versions.

You need to figure out your number of non-zeros per row of you global
matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation.
Otherwise it has to allocate and copy memory often.

You could increase your f9 on a serial run and see what runs best and then
move to parallel with a value in f6 of about 1/2 of f9.

On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> Thanks Mark,
>
>     I use PETSC 3.9.4, is this the optimized version you called?
>
>     Actually f9 and f6 are from the PETSC example. I don't know how to set
> the value correctly so I commend them. The size of my elemental matrix may
> vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or
> some other sizes. And the index is not continuous. In this case, the
> elemental matrix may interlace with each other in the global matrix, and I
> may have thousands of elemental matrix to be assembled. Does the
> preallocating suitable for this?
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Tuesday, January 29, 2019 8:25 PM
> *To:* Yaxiong Chen
> *Cc:* Song Gao; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> Slow assembly is often from not preallocating correctly. I am guessing
> that you are using Q1 element and f9==9, and thus the preallocation should
> be OK if this is a scalar problem on a regular grid and f6-==6 should be OK
> for the off processor allocation, if my assumptions are correct.
>
> You can run with -info, which will tell you how many allocation were done
> in assembly. Make sure that it is small (eg, 0).
>
> I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy
> things with seeming simple code. You could just do this manually if you can
> find anything else.
>
> And I trust you are running an optimized version of PETSc.
>
>
> On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi Song,
>
> I don't quite understand how I can use this command. I don't partition the
> gloabl matrix. If I add my elemental matrix to the global system it will be
> like this. And in my parallel part, I use each core to generate the
> elemental matrix in turn. In this case, I guess each core will be assigned
> the space for global matrix and finally be assembled.But according to the
> manual, it seems each core will store a part of the global matrix. Is the local
> submatrix in the MatMPIAIJSetPreallocation
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
> (Mat
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat>
> B,PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nnz[],PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nnz[])the same as my elemental matrix?
>
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Song Gao <song.gao2 at mail.mcgill.ca>
> *Sent:* Tuesday, January 29, 2019 1:22 PM
> *To:* Yaxiong Chen
> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> I think you would prefer to preallocate the matrix
>
> uncomment this line
> ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>
>
>
> Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <
> petsc-users at mcs.anl.gov> a ?crit :
>
> Hello,
>
>
> I have a 2D system which is assembled by each elemental matrix.  Ae is my
> elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side,
> idx is the global index. My code is as follow, however ,the assembling rate
> is super slow(Marked red in the code). I am not sure whether the assembling
> type is right or not. Since for each element, idx are not continuous
> numbers. Do you have any idea what is the better way to assemble the matrix?
>
>
> Thanks
>
>
> block
>     PetscErrorCode ierr
>      PetscMPIInt rank,nproc, mystart
>      PetscInt  nelem
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
>      integer, allocatable      :: idx(:)
>      PetscScalar, pointer :: xx_v(:)
>
>      PC               prec
>      PetscScalar ::   val
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>      PetscViewer viewer
>      PetscInt  geq,j,k,ne,M,Istart,Iend
>      PetscBool  flg
>      KSPConvergedReason reason
>      Vec         dummyVec, dummyVecs(1)
>      MatNullSpace nullspace
>
>      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
>
>      if (ierr .ne. 0) then
>      print*,'Unable to initialize PETSc'
>      stop
>      endif
>      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
>      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>      mystart=rank+1
>
>  !   Parameter set
>      info=ptsystem%getInitInfo()
>      nelem =
> info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
>      print*,'nelem',nelem
>  !-------------------------------------
>  !  Create Matrix
>      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
>      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1),
> ierr )
>      call MatSetType( Amat, MATMPIBAIJ, ierr )
>      ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>      call MatSetFromOptions( Amat, ierr )
>      call MatSetUp( Amat, ierr )
>      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
>
>       xvec = tVec(0)
>       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
>       call VecSetFromOptions( xvec, ierr )
>       call VecDuplicate( xvec, bvec, ierr )
>       call VecDuplicate( xvec, uvec, ierr )
>
>    t1 = MPI_WTime();
>
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             if (allocated(auxRHSe))  call
> VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
>              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
>       end do
>
>       nelem   = info(Info_EleRHSNum)
>       mystart = rank+1
>
>       do i = mystart, nelem, nproc
>         call ptSystem%getElementalRHS(i, RHSe, idx)
>         print*,'idx',idx
>         ne=size(idx)
>         if (allocated(RHSe))  call
> VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
>       end do
>       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       !  this part is slow, the for loop above is done but here it may get
> stuck
>       call VecAssemblyBegin(bvec,ierr)
>                                 ! For a 2500 DOF system, assembling only
> takes over 2 seconds
>       call VecAssemblyEnd(bvec,ierr)
>                                 ! But for a 10000 DOF system , it gets stuck
>      t2 = MPI_WTime();
>       print*,'assembeling time',t2-t1
>       ! Solve
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>       call KSPSetOperators(ksp,Amat,Amat,ierr)
>       call KSPSetFromOptions(ksp,ierr)
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     call KSPSetType(ksp,KSPCG,ierr)
>     call KSPGetPC(ksp,prec,ierr)
>    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
>     call PCSetType(prec,PCJACOBI,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>      call PetscFinalize(ierr)
>
> end block
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/077f2f4b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3840 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190129/077f2f4b/attachment-0001.png>

From nahmad16 at ku.edu.tr  Wed Jan 30 05:40:21 2019
From: nahmad16 at ku.edu.tr (Najeeb Ahmad)
Date: Wed, 30 Jan 2019 14:40:21 +0300
Subject: [petsc-users] PETSc GPU
In-Reply-To: <CADOhEh5eH+dnmWRSb-_8-sL-9CRORnPcyPFGMsWA90qCAQ2=AA@mail.gmail.com>
References: <CAJJ73m11jF_gYHrNfHrH9S1SwrrmqRMCfms9Gu1AKV=JTC-9QA@mail.gmail.com>
	<CADOhEh5eH+dnmWRSb-_8-sL-9CRORnPcyPFGMsWA90qCAQ2=AA@mail.gmail.com>
Message-ID: <CAJJ73m0j7rO-OMpdQ2=Mzz7m=-p9z3m5G_rCGGxEbKadm8nLTg@mail.gmail.com>

Thanks Mark. I have checked the link and apparently the options
like seqaijcusparse,mpiaijcusparse indicate that PETSc GPU functions now
use cuSPARSE library. Just wanted to make sure.

On Wed, Jan 30, 2019 at 4:35 AM Mark Adams <mfadams at lbl.gov> wrote:

> https://www.mcs.anl.gov/petsc/features/gpus.html
>
> On Tue, Jan 29, 2019 at 2:38 PM Najeeb Ahmad via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi all,
>>
>> I am interested in knowing the state of the art in PETSc GPU
>> implementation. In the paper "Preliminary Implementation of PETSc Using
>> GPUs", it is mentioned that GPU implementation uses cusp and thrust
>> libraries. Is this currently true as well or does PETSc now use cuSPARSE or
>> any other libraries and/or any other improvements?
>>
>> Best Regards,
>>
>> --
>> *Najeeb Ahmad*
>>
>>
>> *Research and Teaching Assistant*
>> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>>
>> *Computer Science and Engineering*
>> *Ko? University, Istanbul, Turkey*
>>
>

-- 
*Najeeb Ahmad*


*Research and Teaching Assistant*
*PARallel and MultiCORE Computing Laboratory (ParCoreLab) *

*Computer Science and Engineering*
*Ko? University, Istanbul, Turkey*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/29a374c8/attachment.html>

From knepley at gmail.com  Wed Jan 30 06:10:13 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Jan 2019 07:10:13 -0500
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
Message-ID: <CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>

On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> just wondering if its possible to manually specify the parallel
> decomposition of data within PETSc Vecs and Mats? And if so, will the
> FieldSplit preconditioning handle this also?
>
> I have an application (non-PETSc) data layout as:
> DATA[16][4][num_procs][3472]
>
> and if possible I would like to use FieldSplit preconditioning without
> re-arranging this layout.
>

Yes. You need to call


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html

in order to define the splits. Note that if you have nested splits, you
will have to also call it on the
nested PC, which stinks, but we have no other way of getting the
information.

Nested splits can also be handled by implementing the DM interface, but
that is a lot of work. It is,
however, the way we handle it internally because it makes things
much easier and more flexible.

  Thanks,

     Matt


> Currently I have a nested FieldSplit preconditioning with 16 outer field
> splits, and a schur complement field split within each. however this throws
> up an indexing error as FieldSplit assumes that the parallel decomposition
> is contiguous (I think).
>
> [0]PETSC ERROR: #1 ISComplement() line 750 in
> /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
>
> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>
> [0]PETSC ERROR: #3 PCSetUp() line 923 in
> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
>
> Cheers, Dave.
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/790ed4c8/attachment.html>

From huq2090 at gmail.com  Wed Jan 30 07:24:32 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Wed, 30 Jan 2019 07:24:32 -0600
Subject: [petsc-users] Problem in MPI execution
Message-ID: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>

Hello PETSc Developers,

I am trying to run my code with the following commands:
$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre
-pc_hypre_type boomeramg
$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type cholesky
-log_view
$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu
-log_view

It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg) but
for 2nd and 3rd case I got the following error message:

Solving the problem...[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: See
http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must
./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
[0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named
huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
[0]PETSC ERROR: Configure options --with-debugging=no --with-64-bit-indices
--download-hypre --download-mpich
[0]PETSC ERROR: #1 MatGetFactor() line 4415 in
/home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
/home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 932 in
/home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 391 in
/home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 723 in
/home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 main() line 199 in
/home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -log_view
[0]PETSC ERROR: -n 9999999
[0]PETSC ERROR: -pc_type ilu
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0


Any suggestion?
Thanks.
Sincerely,
Huq

-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/5214a5f6/attachment.html>

From knepley at gmail.com  Wed Jan 30 07:32:28 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Jan 2019 08:32:28 -0500
Subject: [petsc-users] Problem in MPI execution
In-Reply-To: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
References: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
Message-ID: <CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>

On Wed, Jan 30, 2019 at 8:25 AM Fazlul Huq via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello PETSc Developers,
>
> I am trying to run my code with the following commands:
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre
> -pc_hypre_type boomeramg
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type
> cholesky -log_view
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu
> -log_view
>

We do not have parallel Cholesky or ILU by default. Here is the table of
solvers: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html

  Matt


> It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg)
> but for 2nd and 3rd case I got the following error message:
>
> Solving the problem...[0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
> possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
> ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> [0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named
> huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
> [0]PETSC ERROR: Configure options --with-debugging=no
> --with-64-bit-indices --download-hypre --download-mpich
> [0]PETSC ERROR: #1 MatGetFactor() line 4415 in
> /home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
> /home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: #3 PCSetUp() line 932 in
> /home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 391 in
> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 723 in
> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #6 main() line 199 in
> /home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -log_view
> [0]PETSC ERROR: -n 9999999
> [0]PETSC ERROR: -pc_type ilu
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0
>
>
> Any suggestion?
> Thanks.
> Sincerely,
> Huq
>
> --
>
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/e1583b3d/attachment-0001.html>

From huq2090 at gmail.com  Wed Jan 30 07:57:13 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Wed, 30 Jan 2019 07:57:13 -0600
Subject: [petsc-users] Problem in MPI execution
In-Reply-To: <CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>
References: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
	<CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>
Message-ID: <CAE5QvP0PxUdOMYCsRBvJN9iusT1-jNm9MLGHaFzbsf+2g=cx1Q@mail.gmail.com>

Thanks Matt.

Is there anyway to go over this problem?
I need to run program with parallel Cholesky and ILU.

Thanks.
Sincerely,
Huq

On Wed, Jan 30, 2019 at 7:32 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 30, 2019 at 8:25 AM Fazlul Huq via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello PETSc Developers,
>>
>> I am trying to run my code with the following commands:
>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre
>> -pc_hypre_type boomeramg
>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type
>> cholesky -log_view
>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu
>> -log_view
>>
>
> We do not have parallel Cholesky or ILU by default. Here is the table of
> solvers:
> https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
>
>   Matt
>
>
>> It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg)
>> but for 2nd and 3rd case I got the following error message:
>>
>> Solving the problem...[0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
>> possible LU and Cholesky solvers
>> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
>> ./configure with --download-<package>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>> [0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named
>> huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
>> [0]PETSC ERROR: Configure options --with-debugging=no
>> --with-64-bit-indices --download-hypre --download-mpich
>> [0]PETSC ERROR: #1 MatGetFactor() line 4415 in
>> /home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
>> /home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
>> [0]PETSC ERROR: #3 PCSetUp() line 932 in
>> /home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #4 KSPSetUp() line 391 in
>> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #5 KSPSolve() line 723 in
>> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #6 main() line 199 in
>> /home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -log_view
>> [0]PETSC ERROR: -n 9999999
>> [0]PETSC ERROR: -pc_type ilu
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0
>>
>>
>> Any suggestion?
>> Thanks.
>> Sincerely,
>> Huq
>>
>> --
>>
>> Fazlul Huq
>> Graduate Research Assistant
>> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
>> University of Illinois at Urbana-Champaign (UIUC)
>> E-mail: huq2090 at gmail.com
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/6f1eb255/attachment.html>

From knepley at gmail.com  Wed Jan 30 09:24:56 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Jan 2019 10:24:56 -0500
Subject: [petsc-users] Problem in MPI execution
In-Reply-To: <CAE5QvP0PxUdOMYCsRBvJN9iusT1-jNm9MLGHaFzbsf+2g=cx1Q@mail.gmail.com>
References: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
	<CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>
	<CAE5QvP0PxUdOMYCsRBvJN9iusT1-jNm9MLGHaFzbsf+2g=cx1Q@mail.gmail.com>
Message-ID: <CAMYG4GkCmTrFhE6rJPksvE=JT1LF973NgMkT6TU1kaZUCSqoCw@mail.gmail.com>

On Wed, Jan 30, 2019 at 8:57 AM Fazlul Huq <huq2090 at gmail.com> wrote:

> Thanks Matt.
>
> Is there anyway to go over this problem?
> I need to run program with parallel Cholesky and ILU.
>

>From the link I sent you, you can try MUMPS and PasTiX for Cholesky.

   Matt


> Thanks.
> Sincerely,
> Huq
>
> On Wed, Jan 30, 2019 at 7:32 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Jan 30, 2019 at 8:25 AM Fazlul Huq via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hello PETSc Developers,
>>>
>>> I am trying to run my code with the following commands:
>>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre
>>> -pc_hypre_type boomeramg
>>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type
>>> cholesky -log_view
>>> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu
>>> -log_view
>>>
>>
>> We do not have parallel Cholesky or ILU by default. Here is the table of
>> solvers:
>> https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
>>
>>   Matt
>>
>>
>>> It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg)
>>> but for 2nd and 3rd case I got the following error message:
>>>
>>> Solving the problem...[0]PETSC ERROR: --------------------- Error
>>> Message --------------------------------------------------------------
>>> [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
>>> possible LU and Cholesky solvers
>>> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
>>> ./configure with --download-<package>
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>>> [0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named
>>> huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
>>> [0]PETSC ERROR: Configure options --with-debugging=no
>>> --with-64-bit-indices --download-hypre --download-mpich
>>> [0]PETSC ERROR: #1 MatGetFactor() line 4415 in
>>> /home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
>>> /home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
>>> [0]PETSC ERROR: #3 PCSetUp() line 932 in
>>> /home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #4 KSPSetUp() line 391 in
>>> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #5 KSPSolve() line 723 in
>>> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: #6 main() line 199 in
>>> /home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -log_view
>>> [0]PETSC ERROR: -n 9999999
>>> [0]PETSC ERROR: -pc_type ilu
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>> application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0
>>>
>>>
>>> Any suggestion?
>>> Thanks.
>>> Sincerely,
>>> Huq
>>>
>>> --
>>>
>>> Fazlul Huq
>>> Graduate Research Assistant
>>> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
>>> University of Illinois at Urbana-Champaign (UIUC)
>>> E-mail: huq2090 at gmail.com
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
>
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190130/4ab2d893/attachment.html>

From davelee2804 at gmail.com  Wed Jan 30 19:36:58 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Thu, 31 Jan 2019 12:36:58 +1100
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
	<CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
Message-ID: <CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>

Thanks Matt,

I'm currently working through the details of the nested field split. I've
decided to re-organise my application data layout so that the local
portions are contiguous in the global space for consistency with the
default PETSc layout.

Does the IndexSet need to be the same size on each processor in order for
this IndexSet to be supplied to a FieldSplit PC?

Moreover can an IndexSet have 0 entries on a particular processor and still
be part of a global FieldSplit PC?

Cheers, Dave.


On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> just wondering if its possible to manually specify the parallel
>> decomposition of data within PETSc Vecs and Mats? And if so, will the
>> FieldSplit preconditioning handle this also?
>>
>> I have an application (non-PETSc) data layout as:
>> DATA[16][4][num_procs][3472]
>>
>> and if possible I would like to use FieldSplit preconditioning without
>> re-arranging this layout.
>>
>
> Yes. You need to call
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
>
> in order to define the splits. Note that if you have nested splits, you
> will have to also call it on the
> nested PC, which stinks, but we have no other way of getting the
> information.
>
> Nested splits can also be handled by implementing the DM interface, but
> that is a lot of work. It is,
> however, the way we handle it internally because it makes things
> much easier and more flexible.
>
>   Thanks,
>
>      Matt
>
>
>> Currently I have a nested FieldSplit preconditioning with 16 outer field
>> splits, and a schur complement field split within each. however this throws
>> up an indexing error as FieldSplit assumes that the parallel decomposition
>> is contiguous (I think).
>>
>> [0]PETSC ERROR: #1 ISComplement() line 750 in
>> /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
>>
>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>
>> [0]PETSC ERROR: #3 PCSetUp() line 923 in
>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
>>
>> Cheers, Dave.
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/f4b8c1e1/attachment.html>

From bsmith at mcs.anl.gov  Wed Jan 30 20:28:33 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 31 Jan 2019 02:28:33 +0000
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
	<CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
	<CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>
Message-ID: <3D711F98-7CAB-4432-9E87-EA984C26B195@anl.gov>



> On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thanks Matt,
> 
> I'm currently working through the details of the nested field split. I've decided to re-organise my application data layout so that the local portions are contiguous in the global space for consistency with the default PETSc layout.
> 
> Does the IndexSet need to be the same size on each processor in order for this IndexSet to be supplied to a FieldSplit PC? 

   No, since it is the local size each process can have a different one.
> 
> Moreover can an IndexSet have 0 entries on a particular processor and still be part of a global FieldSplit PC?

   Yes (at least in theory, hopefully there are not bugs for this corner case).
> 
> Cheers, Dave. 
> 
> 
> On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi,
> 
> just wondering if its possible to manually specify the parallel decomposition of data within PETSc Vecs and Mats? And if so, will the FieldSplit preconditioning handle this also?
> 
> I have an application (non-PETSc) data layout as:
> DATA[16][4][num_procs][3472]
> 
> and if possible I would like to use FieldSplit preconditioning without re-arranging this layout.
> 
> Yes. You need to call
> 
>   https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
> 
> in order to define the splits. Note that if you have nested splits, you will have to also call it on the
> nested PC, which stinks, but we have no other way of getting the information.
> 
> Nested splits can also be handled by implementing the DM interface, but that is a lot of work. It is,
> however, the way we handle it internally because it makes things much easier and more flexible.
> 
>   Thanks,
> 
>      Matt
>  
> Currently I have a nested FieldSplit preconditioning with 16 outer field splits, and a schur complement field split within each. however this throws up an indexing error as FieldSplit assumes that the parallel decomposition is contiguous (I think).
> 
> [0]PETSC ERROR: #1 ISComplement() line 750 in /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
> 
> Cheers, Dave.
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From bsmith at mcs.anl.gov  Wed Jan 30 20:33:36 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 31 Jan 2019 02:33:36 +0000
Subject: [petsc-users] Problem in MPI execution
In-Reply-To: <CAMYG4GkCmTrFhE6rJPksvE=JT1LF973NgMkT6TU1kaZUCSqoCw@mail.gmail.com>
References: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
	<CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>
	<CAE5QvP0PxUdOMYCsRBvJN9iusT1-jNm9MLGHaFzbsf+2g=cx1Q@mail.gmail.com>
	<CAMYG4GkCmTrFhE6rJPksvE=JT1LF973NgMkT6TU1kaZUCSqoCw@mail.gmail.com>
Message-ID: <5EB6EF10-BDFF-4B53-9C6F-8F2F8F68BE75@anl.gov>



> On Jan 30, 2019, at 9:24 AM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> On Wed, Jan 30, 2019 at 8:57 AM Fazlul Huq <huq2090 at gmail.com> wrote:
> Thanks Matt.
> 
> Is there anyway to go over this problem?
> I need to run program with parallel Cholesky and ILU.
> 
> From the link I sent you, you can try MUMPS and PasTiX for Cholesky.

   And look at the -pc_hypre_type option; one of them is for ILU.


> 
>    Matt
>  
> Thanks.
> Sincerely,
> Huq
> 
> On Wed, Jan 30, 2019 at 7:32 AM Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Jan 30, 2019 at 8:25 AM Fazlul Huq via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello PETSc Developers,
> 
> I am trying to run my code with the following commands:
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre -pc_hypre_type boomeramg
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type cholesky -log_view
> $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu -log_view
> 
> We do not have parallel Cholesky or ILU by default. Here is the table of solvers: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
> 
>   Matt
>  
> It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg) but for 2nd and 3rd case I got the following error message: 
> 
> Solving the problem...[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018 
> [0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
> [0]PETSC ERROR: Configure options --with-debugging=no --with-64-bit-indices --download-hypre --download-mpich
> [0]PETSC ERROR: #1 MatGetFactor() line 4415 in /home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: #3 PCSetUp() line 932 in /home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 391 in /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 723 in /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #6 main() line 199 in /home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -log_view
> [0]PETSC ERROR: -n 9999999
> [0]PETSC ERROR: -pc_type ilu
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0
> 
> 
> Any suggestion?
> Thanks.
> Sincerely,
> Huq
> 
> -- 
> 
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> 
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From davelee2804 at gmail.com  Wed Jan 30 23:22:24 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Thu, 31 Jan 2019 16:22:24 +1100
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <3D711F98-7CAB-4432-9E87-EA984C26B195@anl.gov>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
	<CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
	<CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>
	<3D711F98-7CAB-4432-9E87-EA984C26B195@anl.gov>
Message-ID: <CAC4fGOGZY7Qsv6fU1AdNGmBiaNmqypP+pVQBiYBCeYNSqgrAcQ@mail.gmail.com>

Hi Barry and Matt,

This seems to work for the outer (composite multiplicative) field split,
but fails for the inner (schur) field split (output error file attached).

I'm setting up the nested field splits as:

  SNESGetKSP(snes, &ksp);

  KSPGetPC(ksp, &pc);

  PCSetType(pc, PCFIELDSPLIT);

  PCFieldSplitSetType(pc, PC_COMPOSITE_MULTIPLICATIVE);


  for(int slice_i = 0; slice_i < nSlice; slice_i++) {

    if(Geometry::procID() == slice_i) {

      ISCreateStride(MPI_COMM_SELF, nDofsSlice, slice_i * nDofsSlice, 1,
&is_s[slice_i]);

    } else {

      ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_s[slice_i]);

    }

    MPI_Barrier(MPI_COMM_WORLD);

    PCFieldSplitSetIS(pc, NULL, is_s[slice_i]);

  }

  PCSetUp(pc);


  PCFieldSplitGetSubKSP(pc, &n_split, &ksp_i);

  for(int slice_i = 0; slice_i < nSlice; slice_i++) {

    KSPGetPC(ksp_i[slice_i], &pc_i);

    PCSetType(pc_i, PCFIELDSPLIT);


    if(Geometry::procID() == slice_i) {

      ISCreateStride(MPI_COMM_SELF, nDofs_u, slice_i * nDofsSlice, 1,
&is_u[slice_i]);

      ISCreateStride(MPI_COMM_SELF, nDofs_p, slice_i * nDofsSlice +
nDofs_u, 1, &is_p[slice_i]);

    } else {

      ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_u[slice_i]);

      ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_p[slice_i]);

    }

    MPI_Barrier(MPI_COMM_WORLD);

    PCFieldSplitSetIS(pc_i, "u", is_u[slice_i]);

    PCFieldSplitSetIS(pc_i, "p", is_p[slice_i]);


    PCFieldSplitSetType(pc_i, PC_COMPOSITE_SCHUR);

    PCSetUp(pc_i);


    if(Geometry::procID() == slice_i) {

      PCFieldSplitGetSubKSP(pc_i, &m_split, &ksp_j);

      KSPSetType(ksp_j[0], KSPGMRES);

      KSPSetType(ksp_j[1], KSPGMRES);

      KSPGetPC(ksp_j[0], &pc_j);

      PCSetType(pc_j, PCJACOBI);

      KSPGetPC(ksp_j[1], &pc_j);

      PCSetType(pc_j, PCJACOBI);

    }

    MPI_Barrier(MPI_COMM_WORLD);

  }


I could re-arrange the indexing and then use PCFieldSplitSetBlockSize(),
however this will undermine the existing block diagonal structure of the
system.

Cheers, Dave.

On Thu, Jan 31, 2019 at 1:28 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Thanks Matt,
> >
> > I'm currently working through the details of the nested field split.
> I've decided to re-organise my application data layout so that the local
> portions are contiguous in the global space for consistency with the
> default PETSc layout.
> >
> > Does the IndexSet need to be the same size on each processor in order
> for this IndexSet to be supplied to a FieldSplit PC?
>
>    No, since it is the local size each process can have a different one.
> >
> > Moreover can an IndexSet have 0 entries on a particular processor and
> still be part of a global FieldSplit PC?
>
>    Yes (at least in theory, hopefully there are not bugs for this corner
> case).
> >
> > Cheers, Dave.
> >
> >
> > On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi,
> >
> > just wondering if its possible to manually specify the parallel
> decomposition of data within PETSc Vecs and Mats? And if so, will the
> FieldSplit preconditioning handle this also?
> >
> > I have an application (non-PETSc) data layout as:
> > DATA[16][4][num_procs][3472]
> >
> > and if possible I would like to use FieldSplit preconditioning without
> re-arranging this layout.
> >
> > Yes. You need to call
> >
> >
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
> >
> > in order to define the splits. Note that if you have nested splits, you
> will have to also call it on the
> > nested PC, which stinks, but we have no other way of getting the
> information.
> >
> > Nested splits can also be handled by implementing the DM interface, but
> that is a lot of work. It is,
> > however, the way we handle it internally because it makes things much
> easier and more flexible.
> >
> >   Thanks,
> >
> >      Matt
> >
> > Currently I have a nested FieldSplit preconditioning with 16 outer field
> splits, and a schur complement field split within each. however this throws
> up an indexing error as FieldSplit assumes that the parallel decomposition
> is contiguous (I think).
> >
> > [0]PETSC ERROR: #1 ISComplement() line 750 in
> /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
> > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > [0]PETSC ERROR: #3 PCSetUp() line 923 in
> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
> >
> > Cheers, Dave.
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/9fdbb74d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: errorOutput.log
Type: application/octet-stream
Size: 38670 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/9fdbb74d/attachment-0001.obj>

From knepley at gmail.com  Thu Jan 31 05:08:05 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Jan 2019 06:08:05 -0500
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <CAC4fGOGZY7Qsv6fU1AdNGmBiaNmqypP+pVQBiYBCeYNSqgrAcQ@mail.gmail.com>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
	<CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
	<CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>
	<3D711F98-7CAB-4432-9E87-EA984C26B195@anl.gov>
	<CAC4fGOGZY7Qsv6fU1AdNGmBiaNmqypP+pVQBiYBCeYNSqgrAcQ@mail.gmail.com>
Message-ID: <CAMYG4G=S+sV_rT=_wTU4gb_r4V+oqD269TqF-Ptu0NirEuo7Cg@mail.gmail.com>

On Thu, Jan 31, 2019 at 12:22 AM Dave Lee <davelee2804 at gmail.com> wrote:

> Hi Barry and Matt,
>
> This seems to work for the outer (composite multiplicative) field split,
> but fails for the inner (schur) field split (output error file attached).
>

The inner FS numbering is with respect to the reduced problem, not the
original global problem (it can't know you did one FS on the outside).

  THanks,

    Matt


> I'm setting up the nested field splits as:
>
>   SNESGetKSP(snes, &ksp);
>
>   KSPGetPC(ksp, &pc);
>
>   PCSetType(pc, PCFIELDSPLIT);
>
>   PCFieldSplitSetType(pc, PC_COMPOSITE_MULTIPLICATIVE);
>
>
>   for(int slice_i = 0; slice_i < nSlice; slice_i++) {
>
>     if(Geometry::procID() == slice_i) {
>
>       ISCreateStride(MPI_COMM_SELF, nDofsSlice, slice_i * nDofsSlice, 1,
> &is_s[slice_i]);
>
>     } else {
>
>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_s[slice_i]);
>
>     }
>
>     MPI_Barrier(MPI_COMM_WORLD);
>
>     PCFieldSplitSetIS(pc, NULL, is_s[slice_i]);
>
>   }
>
>   PCSetUp(pc);
>
>
>   PCFieldSplitGetSubKSP(pc, &n_split, &ksp_i);
>
>   for(int slice_i = 0; slice_i < nSlice; slice_i++) {
>
>     KSPGetPC(ksp_i[slice_i], &pc_i);
>
>     PCSetType(pc_i, PCFIELDSPLIT);
>
>
>     if(Geometry::procID() == slice_i) {
>
>       ISCreateStride(MPI_COMM_SELF, nDofs_u, slice_i * nDofsSlice, 1,
> &is_u[slice_i]);
>
>       ISCreateStride(MPI_COMM_SELF, nDofs_p, slice_i * nDofsSlice +
> nDofs_u, 1, &is_p[slice_i]);
>
>     } else {
>
>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_u[slice_i]);
>
>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_p[slice_i]);
>
>     }
>
>     MPI_Barrier(MPI_COMM_WORLD);
>
>     PCFieldSplitSetIS(pc_i, "u", is_u[slice_i]);
>
>     PCFieldSplitSetIS(pc_i, "p", is_p[slice_i]);
>
>
>     PCFieldSplitSetType(pc_i, PC_COMPOSITE_SCHUR);
>
>     PCSetUp(pc_i);
>
>
>     if(Geometry::procID() == slice_i) {
>
>       PCFieldSplitGetSubKSP(pc_i, &m_split, &ksp_j);
>
>       KSPSetType(ksp_j[0], KSPGMRES);
>
>       KSPSetType(ksp_j[1], KSPGMRES);
>
>       KSPGetPC(ksp_j[0], &pc_j);
>
>       PCSetType(pc_j, PCJACOBI);
>
>       KSPGetPC(ksp_j[1], &pc_j);
>
>       PCSetType(pc_j, PCJACOBI);
>
>     }
>
>     MPI_Barrier(MPI_COMM_WORLD);
>
>   }
>
>
> I could re-arrange the indexing and then use PCFieldSplitSetBlockSize(),
> however this will undermine the existing block diagonal structure of the
> system.
>
> Cheers, Dave.
>
> On Thu, Jan 31, 2019 at 1:28 PM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>>
>> > On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Thanks Matt,
>> >
>> > I'm currently working through the details of the nested field split.
>> I've decided to re-organise my application data layout so that the local
>> portions are contiguous in the global space for consistency with the
>> default PETSc layout.
>> >
>> > Does the IndexSet need to be the same size on each processor in order
>> for this IndexSet to be supplied to a FieldSplit PC?
>>
>>    No, since it is the local size each process can have a different one.
>> >
>> > Moreover can an IndexSet have 0 entries on a particular processor and
>> still be part of a global FieldSplit PC?
>>
>>    Yes (at least in theory, hopefully there are not bugs for this corner
>> case).
>> >
>> > Cheers, Dave.
>> >
>> >
>> > On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > Hi,
>> >
>> > just wondering if its possible to manually specify the parallel
>> decomposition of data within PETSc Vecs and Mats? And if so, will the
>> FieldSplit preconditioning handle this also?
>> >
>> > I have an application (non-PETSc) data layout as:
>> > DATA[16][4][num_procs][3472]
>> >
>> > and if possible I would like to use FieldSplit preconditioning without
>> re-arranging this layout.
>> >
>> > Yes. You need to call
>> >
>> >
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
>> >
>> > in order to define the splits. Note that if you have nested splits, you
>> will have to also call it on the
>> > nested PC, which stinks, but we have no other way of getting the
>> information.
>> >
>> > Nested splits can also be handled by implementing the DM interface, but
>> that is a lot of work. It is,
>> > however, the way we handle it internally because it makes things much
>> easier and more flexible.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > Currently I have a nested FieldSplit preconditioning with 16 outer
>> field splits, and a schur complement field split within each. however this
>> throws up an indexing error as FieldSplit assumes that the parallel
>> decomposition is contiguous (I think).
>> >
>> > [0]PETSC ERROR: #1 ISComplement() line 750 in
>> /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
>> > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> > [0]PETSC ERROR: #3 PCSetUp() line 923 in
>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
>> >
>> > Cheers, Dave.
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/4431c93e/attachment.html>

From rickcha at googlemail.com  Thu Jan 31 05:20:59 2019
From: rickcha at googlemail.com (rickcha at googlemail.com)
Date: Thu, 31 Jan 2019 12:20:59 +0100
Subject: [petsc-users] DMGetStratumSize smaller than expected
Message-ID: <5B01596D-4E86-40D5-8074-74E8E5DB0BE2@googlemail.com>

Hello,

I have an issue with DMGetStatumSize() and DMGetStratumIS().

My intention is to access all vertices (height = 0) that are part of a named label in my DM.

The problem is that only vertices that are exclusively part of this named label seem to be included. The label  is assigned to a face, let?s call it ?A? for simplicities sake. Adjacent faces have their own label (and Dirichlet BC?s prescribed to them if that matters) which I will refer to as (?B?,?C?,?D? and ?E?). Vertices that lie on the edge should be part of more than one stratum. I.e. belonging to ?A? AND ?B?, or ?A? AND ?B? AND ?C? if they lie on a corner where the three faces meet.
However, those shared vertices seem not to be included when calling DMGetStatumSize() and DMGetStratumIS().

For a 5x5x5 hex cube with faces labeled ?A?, ?B?, [?], I would expect DMGetStratumSize(dm,?A?,0,&size) to yield size = 36. But I get 16 instead, missing the 20 ?shared? vertices.

Is there a way to change this behaviour? Or do I have to get the face stratum (height = 2) and then refer to the cone of the cone to get what I want?


Thanks,
Max

From knepley at gmail.com  Thu Jan 31 05:50:50 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Jan 2019 06:50:50 -0500
Subject: [petsc-users] DMGetStratumSize smaller than expected
In-Reply-To: <5B01596D-4E86-40D5-8074-74E8E5DB0BE2@googlemail.com>
References: <5B01596D-4E86-40D5-8074-74E8E5DB0BE2@googlemail.com>
Message-ID: <CAMYG4GnrnerejCcOxObqvTLEtVry-MzTReJ-UiKjp-PWBd3vsw@mail.gmail.com>

On Thu, Jan 31, 2019 at 6:22 AM rickcha--- via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I have an issue with DMGetStatumSize() and DMGetStratumIS().
>
> My intention is to access all vertices (height = 0) that are part of a
> named label in my DM.
>

Depth = 0 are the vertices, but this works with DMPlexGetDpethStratum().
You seems like
you are calling DMLabelGetStratumIS(). The value here is the label value,
not the height or depth.

  Thanks,

     Matt


> The problem is that only vertices that are exclusively part of this named
> label seem to be included. The label  is assigned to a face, let?s call it
> ?A? for simplicities sake. Adjacent faces have their own label (and
> Dirichlet BC?s prescribed to them if that matters) which I will refer to as
> (?B?,?C?,?D? and ?E?). Vertices that lie on the edge should be part of more
> than one stratum. I.e. belonging to ?A? AND ?B?, or ?A? AND ?B? AND ?C? if
> they lie on a corner where the three faces meet.
> However, those shared vertices seem not to be included when calling
> DMGetStatumSize() and DMGetStratumIS().
>
> For a 5x5x5 hex cube with faces labeled ?A?, ?B?, [?], I would expect
> DMGetStratumSize(dm,?A?,0,&size) to yield size = 36. But I get 16 instead,
> missing the 20 ?shared? vertices.
>
> Is there a way to change this behaviour? Or do I have to get the face
> stratum (height = 2) and then refer to the cone of the cone to get what I
> want?
>
>
> Thanks,
> Max



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/ab0a250a/attachment-0001.html>

From jychang48 at gmail.com  Thu Jan 31 14:18:26 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 31 Jan 2019 13:18:26 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
Message-ID: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>

Hi all,

I'm working with some folks to extract a linear system of equations from an
external software package that solves power flow equations in complex form.
Since that external package uses serial direct solvers like KLU from
suitesparse, I want a proof-of-concept where the same matrix can be solved
in PETSc using its parallel solvers.

I got mumps to achieve a very minor speedup across two MPI processes on a
single node (went from solving a 300k dog system in 1.8 seconds to 1.5
seconds). However I want to use iterative solvers and preconditioners but I
have never worked with complex numbers so I am not sure what the "best"
options are given PETSc's capabilities.

So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I
also tried BICG with BJACOBI and while it did converge it converged slowly.
Does anyone have recommendations on how one would go about preconditioning
PETSc matrices with complex numbers? I was originally thinking about
converting it to cartesian form: Declaring all voltages =
sqrt(real^2+imaginary^2) and all angles to be something like a conditional
arctan(imaginary/real) because all the papers I've seen in literature that
claim to successfully precondition power flow equations operate in this
form.

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/0079fed8/attachment.html>

From knepley at gmail.com  Thu Jan 31 14:46:47 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Jan 2019 15:46:47 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
Message-ID: <CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>

On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi all,
>
> I'm working with some folks to extract a linear system of equations from
> an external software package that solves power flow equations in complex
> form. Since that external package uses serial direct solvers like KLU from
> suitesparse, I want a proof-of-concept where the same matrix can be solved
> in PETSc using its parallel solvers.
>
> I got mumps to achieve a very minor speedup across two MPI processes on a
> single node (went from solving a 300k dog system in 1.8 seconds to 1.5
> seconds). However I want to use iterative solvers and preconditioners but I
> have never worked with complex numbers so I am not sure what the "best"
> options are given PETSc's capabilities.
>
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe
> I also tried BICG with BJACOBI and while it did converge it converged
> slowly. Does anyone have recommendations on how one would go about
> preconditioning PETSc matrices with complex numbers? I was originally
> thinking about converting it to cartesian form: Declaring all voltages =
> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
> arctan(imaginary/real) because all the papers I've seen in literature that
> claim to successfully precondition power flow equations operate in this
> form.
>

1) We really need to see the (simplified) equations

2) All complex equations can be converted to a system of real equations
twice as large, but this is not necessarily the best way to go

 Thanks,

    Matt


> Justin
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/f50b2a73/attachment.html>

From jychang48 at gmail.com  Thu Jan 31 17:21:28 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 31 Jan 2019 16:21:28 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
Message-ID: <CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>

Here's IMHO the simplest explanation of the equations I'm trying to solve:

http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf

Right now we're just trying to solve eq(5) (in section 1), inverting the
linear Y-bus matrix. Eventually we have to be able to solve equations like
those in the next section.

On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi all,
>>
>> I'm working with some folks to extract a linear system of equations from
>> an external software package that solves power flow equations in complex
>> form. Since that external package uses serial direct solvers like KLU from
>> suitesparse, I want a proof-of-concept where the same matrix can be solved
>> in PETSc using its parallel solvers.
>>
>> I got mumps to achieve a very minor speedup across two MPI processes on a
>> single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>> seconds). However I want to use iterative solvers and preconditioners but I
>> have never worked with complex numbers so I am not sure what the "best"
>> options are given PETSc's capabilities.
>>
>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe
>> I also tried BICG with BJACOBI and while it did converge it converged
>> slowly. Does anyone have recommendations on how one would go about
>> preconditioning PETSc matrices with complex numbers? I was originally
>> thinking about converting it to cartesian form: Declaring all voltages =
>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>> arctan(imaginary/real) because all the papers I've seen in literature that
>> claim to successfully precondition power flow equations operate in this
>> form.
>>
>
> 1) We really need to see the (simplified) equations
>
> 2) All complex equations can be converted to a system of real equations
> twice as large, but this is not necessarily the best way to go
>
>  Thanks,
>
>     Matt
>
>
>> Justin
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/350444d5/attachment.html>

From knepley at gmail.com  Thu Jan 31 18:00:25 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Jan 2019 19:00:25 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
Message-ID: <CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>

On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:

> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>
> Right now we're just trying to solve eq(5) (in section 1), inverting the
> linear Y-bus matrix. Eventually we have to be able to solve equations like
> those in the next section.
>

Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix
to me (if all the y's are positive). This means
that it should be really easy to solve, and I think GAMG should do it. You
can start out just doing relaxation, like SOR, on
small examples.

  Thanks,

    Matt


> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi all,
>>>
>>> I'm working with some folks to extract a linear system of equations from
>>> an external software package that solves power flow equations in complex
>>> form. Since that external package uses serial direct solvers like KLU from
>>> suitesparse, I want a proof-of-concept where the same matrix can be solved
>>> in PETSc using its parallel solvers.
>>>
>>> I got mumps to achieve a very minor speedup across two MPI processes on
>>> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>>> seconds). However I want to use iterative solvers and preconditioners but I
>>> have never worked with complex numbers so I am not sure what the "best"
>>> options are given PETSc's capabilities.
>>>
>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>> believe I also tried BICG with BJACOBI and while it did converge it
>>> converged slowly. Does anyone have recommendations on how one would go
>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>> thinking about converting it to cartesian form: Declaring all voltages =
>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>> claim to successfully precondition power flow equations operate in this
>>> form.
>>>
>>
>> 1) We really need to see the (simplified) equations
>>
>> 2) All complex equations can be converted to a system of real equations
>> twice as large, but this is not necessarily the best way to go
>>
>>  Thanks,
>>
>>     Matt
>>
>>
>>> Justin
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/4e0c10f0/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Thu Jan 31 19:07:49 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 31 Jan 2019 19:07:49 -0600
Subject: [petsc-users] petsc4py for numpy array <-> MATDENSE binary
Message-ID: <CAOGsD9gWWTCGAJVMgBs9rgF16uAUY_XGqBqf02=xzXLz+yFsiQ@mail.gmail.com>

Hi,

I have a large (~25k x 25k x 50) 3D array that I want to store as binary
PETSc readable matrix using the MATDENSE format. I've currently achieved
this (for smaller arrays) by writing the matrix out to an ASCII file and
converting this ASCII file to binary, one number at a time using a C
script.

Does petsc4py support converting a 3d numpy array (complex valued) to
MATDENSE (MPI) binary format directly ?

I saw some discussion on this topic back in 2012 on the mailing list but
it's not clear to me what happened after that.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190131/e2c897f8/attachment-0001.html>

From davelee2804 at gmail.com  Thu Jan 31 19:30:32 2019
From: davelee2804 at gmail.com (Dave Lee)
Date: Fri, 1 Feb 2019 12:30:32 +1100
Subject: [petsc-users] Vecs and Mats with non-contiguous parallel layout
In-Reply-To: <CAMYG4G=S+sV_rT=_wTU4gb_r4V+oqD269TqF-Ptu0NirEuo7Cg@mail.gmail.com>
References: <CAC4fGOEDj7TQkkYLfwDKLhF8wn+atGj=-1drJ259bp8h=PBZWg@mail.gmail.com>
	<CAMYG4Gkcv86OaR5Y0da_wfeovhW-g3wvwr8BykWAPbCSNYwOzg@mail.gmail.com>
	<CAC4fGOG7vSY+s_tRp7EQ8FjhqseZU_e14zOqK+JiuxAkq9G_QA@mail.gmail.com>
	<3D711F98-7CAB-4432-9E87-EA984C26B195@anl.gov>
	<CAC4fGOGZY7Qsv6fU1AdNGmBiaNmqypP+pVQBiYBCeYNSqgrAcQ@mail.gmail.com>
	<CAMYG4G=S+sV_rT=_wTU4gb_r4V+oqD269TqF-Ptu0NirEuo7Cg@mail.gmail.com>
Message-ID: <CAC4fGOHZO-nj0k7=qvgHDbmKRioM7REmQ5SYp_NN5r2YbPssaA@mail.gmail.com>

Whoops, thanks Matt, well spotted. Looks a lot better now.

Cheers, Dave.

On Thu, Jan 31, 2019 at 10:08 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 31, 2019 at 12:22 AM Dave Lee <davelee2804 at gmail.com> wrote:
>
>> Hi Barry and Matt,
>>
>> This seems to work for the outer (composite multiplicative) field split,
>> but fails for the inner (schur) field split (output error file attached).
>>
>
> The inner FS numbering is with respect to the reduced problem, not the
> original global problem (it can't know you did one FS on the outside).
>
>   THanks,
>
>     Matt
>
>
>> I'm setting up the nested field splits as:
>>
>>   SNESGetKSP(snes, &ksp);
>>
>>   KSPGetPC(ksp, &pc);
>>
>>   PCSetType(pc, PCFIELDSPLIT);
>>
>>   PCFieldSplitSetType(pc, PC_COMPOSITE_MULTIPLICATIVE);
>>
>>
>>   for(int slice_i = 0; slice_i < nSlice; slice_i++) {
>>
>>     if(Geometry::procID() == slice_i) {
>>
>>       ISCreateStride(MPI_COMM_SELF, nDofsSlice, slice_i * nDofsSlice, 1,
>> &is_s[slice_i]);
>>
>>     } else {
>>
>>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_s[slice_i]);
>>
>>     }
>>
>>     MPI_Barrier(MPI_COMM_WORLD);
>>
>>     PCFieldSplitSetIS(pc, NULL, is_s[slice_i]);
>>
>>   }
>>
>>   PCSetUp(pc);
>>
>>
>>   PCFieldSplitGetSubKSP(pc, &n_split, &ksp_i);
>>
>>   for(int slice_i = 0; slice_i < nSlice; slice_i++) {
>>
>>     KSPGetPC(ksp_i[slice_i], &pc_i);
>>
>>     PCSetType(pc_i, PCFIELDSPLIT);
>>
>>
>>     if(Geometry::procID() == slice_i) {
>>
>>       ISCreateStride(MPI_COMM_SELF, nDofs_u, slice_i * nDofsSlice, 1,
>> &is_u[slice_i]);
>>
>>       ISCreateStride(MPI_COMM_SELF, nDofs_p, slice_i * nDofsSlice +
>> nDofs_u, 1, &is_p[slice_i]);
>>
>>     } else {
>>
>>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_u[slice_i]);
>>
>>       ISCreateStride(MPI_COMM_SELF, 0, 0, 1, &is_p[slice_i]);
>>
>>     }
>>
>>     MPI_Barrier(MPI_COMM_WORLD);
>>
>>     PCFieldSplitSetIS(pc_i, "u", is_u[slice_i]);
>>
>>     PCFieldSplitSetIS(pc_i, "p", is_p[slice_i]);
>>
>>
>>     PCFieldSplitSetType(pc_i, PC_COMPOSITE_SCHUR);
>>
>>     PCSetUp(pc_i);
>>
>>
>>     if(Geometry::procID() == slice_i) {
>>
>>       PCFieldSplitGetSubKSP(pc_i, &m_split, &ksp_j);
>>
>>       KSPSetType(ksp_j[0], KSPGMRES);
>>
>>       KSPSetType(ksp_j[1], KSPGMRES);
>>
>>       KSPGetPC(ksp_j[0], &pc_j);
>>
>>       PCSetType(pc_j, PCJACOBI);
>>
>>       KSPGetPC(ksp_j[1], &pc_j);
>>
>>       PCSetType(pc_j, PCJACOBI);
>>
>>     }
>>
>>     MPI_Barrier(MPI_COMM_WORLD);
>>
>>   }
>>
>>
>> I could re-arrange the indexing and then use PCFieldSplitSetBlockSize(),
>> however this will undermine the existing block diagonal structure of the
>> system.
>>
>> Cheers, Dave.
>>
>> On Thu, Jan 31, 2019 at 1:28 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>> wrote:
>>
>>>
>>>
>>> > On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >
>>> > Thanks Matt,
>>> >
>>> > I'm currently working through the details of the nested field split.
>>> I've decided to re-organise my application data layout so that the local
>>> portions are contiguous in the global space for consistency with the
>>> default PETSc layout.
>>> >
>>> > Does the IndexSet need to be the same size on each processor in order
>>> for this IndexSet to be supplied to a FieldSplit PC?
>>>
>>>    No, since it is the local size each process can have a different one.
>>> >
>>> > Moreover can an IndexSet have 0 entries on a particular processor and
>>> still be part of a global FieldSplit PC?
>>>
>>>    Yes (at least in theory, hopefully there are not bugs for this corner
>>> case).
>>> >
>>> > Cheers, Dave.
>>> >
>>> >
>>> > On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> > Hi,
>>> >
>>> > just wondering if its possible to manually specify the parallel
>>> decomposition of data within PETSc Vecs and Mats? And if so, will the
>>> FieldSplit preconditioning handle this also?
>>> >
>>> > I have an application (non-PETSc) data layout as:
>>> > DATA[16][4][num_procs][3472]
>>> >
>>> > and if possible I would like to use FieldSplit preconditioning without
>>> re-arranging this layout.
>>> >
>>> > Yes. You need to call
>>> >
>>> >
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
>>> >
>>> > in order to define the splits. Note that if you have nested splits,
>>> you will have to also call it on the
>>> > nested PC, which stinks, but we have no other way of getting the
>>> information.
>>> >
>>> > Nested splits can also be handled by implementing the DM interface,
>>> but that is a lot of work. It is,
>>> > however, the way we handle it internally because it makes things much
>>> easier and more flexible.
>>> >
>>> >   Thanks,
>>> >
>>> >      Matt
>>> >
>>> > Currently I have a nested FieldSplit preconditioning with 16 outer
>>> field splits, and a schur complement field split within each. however this
>>> throws up an indexing error as FieldSplit assumes that the parallel
>>> decomposition is contiguous (I think).
>>> >
>>> > [0]PETSC ERROR: #1 ISComplement() line 750 in
>>> /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
>>> > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in
>>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>>> > [0]PETSC ERROR: #3 PCSetUp() line 923 in
>>> /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
>>> >
>>> > Cheers, Dave.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/ac81e6b6/attachment.html>

From jed at jedbrown.org  Thu Jan 31 22:28:16 2019
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 31 Jan 2019 21:28:16 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
Message-ID: <87sgx8f7z3.fsf@jedbrown.org>

Justin Chang via petsc-users <petsc-users at mcs.anl.gov> writes:

> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>
> Right now we're just trying to solve eq(5) (in section 1), inverting the
> linear Y-bus matrix. Eventually we have to be able to solve equations like
> those in the next section.

It's always useful to pick a smallish problem and compute the spectrum
of the operator and plot the eigenvectors associated with interesting
eigenvalues (especially those near zero and those with largest
magnitude).  Eq 5 looks like a graph Laplacian.