[petsc-users] Nullspaces

Tue Jan 4 09:56:26 CST 2022

Hello Mark,
I analyzed the codes with valgrind, both the real code and the tiny one.
I obviously used memcheck tool but with full leak check compiling the codes with debug info.
Not considering OpenMPI events (I have no wrappers on the machine I used for the analysis), the real code gave zero errors and the tiny one gave this
==17911== 905,536 (1,552 direct, 903,984 indirect) bytes in 1 blocks are definitely lost in loss record 33 of 33
==17911==    at 0x483E340: memalign (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==17911==    by 0x49CB672: PetscMallocAlign (in /usr/lib/x86_64-linux-gnu/libpetsc_real.so.3.12.4)
==17911==    by 0x49CBE1D: PetscMallocA (in /usr/lib/x86_64-linux-gnu/libpetsc_real.so.3.12.4)
==17911==    by 0x4B26187: VecCreate (in /usr/lib/x86_64-linux-gnu/libpetsc_real.so.3.12.4)
==17911==    by 0x10940D: main (testNullSpace.cpp:30)

due to the fact that I forgot to destroy the solution Vec (adding VecDestroy(&solution) at the end of the main, the error disappear).
For both the codes, I analyzed the two ways of passing the constant to the null space of the operator, no memory errors but still the same results from MatNullSpaceTest, i.e.

MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, nullptr, &nullspace);

passes the test while

Vec* nsp;
VecDuplicateVecs(solution, 1, &nsp);
VecSet(nsp[0],1.0);
VecNormalize(nsp[0], nullptr);
MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_FALSE, 1, nsp, &nullspace);
VecDestroyVecs(1,&nsp);
PetscFree(nsp);

does not.

I hope this can satisfy your doubt about the memory behavior, but please do not hesitate to ask for more analysis if it cannot.

As Matthew said some weeks ago, something should be wrong in the code, I would say in the matrix, that’s why I provided the matrix and the way I test it.
Unfortunately, it is hard (read impossible) for me to share the code producing the matrix. I hope the minimal code I provided is enough to understand something.

Thank you all.

Marco Cisternino

From: Marco Cisternino
Sent: lunedì 3 gennaio 2022 16:08
To: Mark Adams <mfadams at lbl.gov>
Cc: Matthew Knepley <knepley at gmail.com>; petsc-users <petsc-users at mcs.anl.gov>
Subject: RE: [petsc-users] Nullspaces

We usually analyze the code with valgrind, when important changes are implemented.
I have to admit that this analysis is still not automatic and the case we are talking about is not a test case for our workload.
The test cases we have give no errors in valgrind analysis.

However, I will analyze both the real code and the tiny one for this case with valgrind and report the results.

Thank you,

Marco Cisternino

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: lunedì 3 gennaio 2022 15:50
To: Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Nullspaces

I have not looked at your code, but as a general observation you want to have some sort of memory checker, like valgrid for CPUs, in your workflow.
It is the fastest way to find some classes of bugs.

On Mon, Jan 3, 2022 at 8:47 AM Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>> wrote:
Are you talking about the code that produce the linear system or about the tiny code that test the null space?
In the first case, it is absolutely possible, but I would expect no problem in the tiny code, do you agree?
It is important to remark that the real code and the tiny one behave in the same way when testing the null space of the operator. I can analyze with valgrind and I will, but I would not expect great insights.

Thanks,

Marco Cisternino, PhD
marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>
______________________
Optimad Engineering Srl
Via Bligny 5, Torino, Italia.
+3901119719782
www.optimad.it<http://www.optimad.it/>

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: lunedì 3 gennaio 2022 14:42
To: Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Nullspaces

There could be a memory bug that does not cause a noticeable problem until it hits some vital data and valgrind might find it on a small problem.

However you might have a bug like a hardwired buffer size that overflows that is in fact not a bug until you get to this large size and in that case valgrid would need to be run on the large case and would have a good chance of finding it.

On Mon, Jan 3, 2022 at 4:42 AM Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>> wrote:
My comments are between the Mark’s lines and they starts with “#”

Marco Cisternino

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: sabato 25 dicembre 2021 14:59
To: Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Nullspaces

If  "triggering the issue" requires a substantial mesh, that makes me think there is a logic bug somewhere. Maybe use valgrind.

# Are you suggesting to use valgrind on this tiny toy code or on the original one? However, considering the purpose of the tiny code, i.e. testing the constant null space, why there should be a logical bug? Case 1 passes and case 2 should be exactly the same, shouldn’t be it?

Also you say you divide by the cell volume. Maybe I am not understanding this but that is basically diagonal scaling and that will change the null space (ie, not a constant anymore)

# I agree on this, but it pushes a question: why the case 1 passes the test?
# Thank you, Mark.

On Thu, Dec 16, 2021 at 11:11 AM Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>> wrote:
Hello Matthew,
as promised I prepared a minimal (112960 rows. I’m not able to produce anything smaller than this and triggering the issue) example of the behavior I was talking about some days ago.
What I did is to produce matrix, right hand side and initial solution of the linear system.

As I told you before, this linear system is the discretization of the pressure equation of a predictor-corrector method for NS equations in the framework of finite volume method.
This case has homogeneous Neumann boundary conditions. Computational domain has two independent and separated sub-domains.
I discretize the weak formulation and I divide every row of the linear system by the volume of the relative cell.
The underlying mesh is not uniform, therefore cells have different volumes.
The issue I’m going to explain does not show up if the mesh is uniform, same volume for all the cells.

I usually build the null space sub-domain by sub-domain with
MatNullSpaceCreate(getCommunicator(), PETSC_FALSE, nConstants, constants, &nullspace);
Where nConstants = 2 and constants contains two normalized arrays with constant values on degrees of freedom relative to the associated sub-domain and zeros elsewhere.

However, as a test I tried the constant over the whole domain using 2 alternatives that should produce the same null space:

  1.  MatNullSpaceCreate(getCommunicator(), PETSC_TRUE, 0, nullptr, &nullspace);
  2.  Vec* nsp;

VecDuplicateVecs(solution, 1, &nsp);

VecSet(nsp[0],1.0);

VecNormalize(nsp[0], nullptr);

MatNullSpaceCreate(getCommunicator(), PETSC_FALSE, 1, nsp, &nullspace);

Once I created the null space I test it using:
MatNullSpaceTest(nullspace, m_A, &isNullSpaceValid);

The case 1 pass the test while case 2 don’t.

I have a small code for matrix loading, null spaces creation and testing.
Unfortunately I cannot implement a small code able to produce that linear system.

As attachment you can find an archive containing the matrix, the initial solution (used to manually build the null space) and the rhs (not used in the test code) in binary format.
You can also find the testing code in the same archive.
I used petsc 3.12(gcc+openMPI) and petsc 3.15.2(intelOneAPI) same results.
If the attachment is not delivered, I can share a link to it.

Thanks for any help.

Marco Cisternino

Marco Cisternino, PhD
marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>
______________________
Optimad Engineering Srl
Via Bligny 5, Torino, Italia.
+3901119719782
www.optimad.it<http://www.optimad.it/>

From: Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>>
Sent: martedì 7 dicembre 2021 19:36
To: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Cc: petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Nullspaces

I will, as soon as possible...

Scarica Outlook per Android<https://aka.ms/AAb9ysg>
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Tuesday, December 7, 2021 7:25:43 PM
To: Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>>
Cc: petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Nullspaces

On Tue, Dec 7, 2021 at 11:19 AM Marco Cisternino <marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>> wrote:

Good morning,

I’m still struggling with the Poisson equation with Neumann BCs.

I discretize the equation by finite volume method and I divide every line of the linear system by the volume of the cell. I could avoid this division, but I’m trying to understand.

My mesh is not uniform, i.e. cells have different volumes (it is an octree mesh).

Moreover, in my computational domain there are 2 separated sub-domains.

I build the null space and then I use MatNullSpaceTest to check it.

If I do this:

MatNullSpaceCreate(getCommunicator(), PETSC_TRUE, 0, nullptr, &nullspace);

It works

This produces the normalized constant vector.

If I do this:

Vec nsp;

VecDuplicate(m_rhs, &nsp);

VecSet(nsp,1.0);

VecNormalize(nsp, nullptr);

MatNullSpaceCreate(getCommunicator(), PETSC_FALSE, 1, &nsp, &nullspace);

It does not work

This is also the normalized constant vector.

So you are saying that these two vectors give different results with MatNullSpaceTest()?
Something must be wrong in the code. Can you send a minimal example of this? I will go
through and debug it.

  Thanks,

     Matt

Probably, I have wrong expectations, but should not it be the same?

Thanks

Marco Cisternino, PhD
marco.cisternino at optimad.it<mailto:marco.cisternino at optimad.it>

______________________

Optimad Engineering Srl

Via Bligny 5, Torino, Italia.
+3901119719782
www.optimad.it<http://www.optimad.it/>

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220104/fe21ad43/attachment-0001.html>