[petsc-users] PETSc initialization error
Sam Guo
sam.guo at cd-adapco.com
Fri Jun 26 15:36:07 CDT 2020
One work around for me is to call PetscInitialize once for my entire
program and skip PetscFinalize (since I don't have a good place to call
PetscFinalize before ending the program).
On Fri, Jun 26, 2020 at 1:33 PM Sam Guo <sam.guo at cd-adapco.com> wrote:
> I get the crash after calling Initialize/Finalize multiple times. Junchao
> fixed the bug for serial but parallel still crashes.
>
> On Fri, Jun 26, 2020 at 1:28 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>> Ah, so you get the crash the second time you call PetscInitialize()?
>> That is a problem because we do intend to support that capability (but you
>> much call PetscFinalize() each time also).
>>
>> Barry
>>
>>
>> On Jun 26, 2020, at 3:25 PM, Sam Guo <sam.guo at cd-adapco.com> wrote:
>>
>> Hi Barry,
>> Thanks for the quick response.
>> I will call PetscInitialize once and skip the PetscFinalize for now to
>> avoid the crash. The crash is actually in PetscInitialize, not
>> PetscFinalize.
>>
>> Thanks,
>> Sam
>>
>> On Fri, Jun 26, 2020 at 1:21 PM Barry Smith <bsmith at petsc.dev> wrote:
>>
>>>
>>> Sam,
>>>
>>> You can skip PetscFinalize() so long as you only call
>>> PetscInitialize() once. It is not desirable in general to skip the finalize
>>> because PETSc can't free all its data structures and you cannot see the
>>> PETSc logging information with -log_view but in terms of the code running
>>> correctly you do not need to call PetscFinalize.
>>>
>>> If your code crashes in PetscFinalize() please send the full error
>>> output and we can try to help you debug it.
>>>
>>>
>>> Barry
>>>
>>> On Jun 26, 2020, at 3:14 PM, Sam Guo <sam.guo at cd-adapco.com> wrote:
>>>
>>> To clarify, we have a mpi wrapper (so we can switch to different mpi at
>>> runtime). I compile petsc using our mpi wrapper.
>>> If I just call PETSc initialize once without calling finallize, it is
>>> ok. My question to you is that: can I skip finalize?
>>> Our program calls mpi_finalize at end anyway.
>>>
>>> On Fri, Jun 26, 2020 at 1:09 PM Sam Guo <sam.guo at cd-adapco.com> wrote:
>>>
>>>> Hi Junchao,
>>>> Attached please find the configure.log.
>>>> I also attach the pinit.c which contains your patch (I am currently
>>>> using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the
>>>> serial version. The error now is about the parallel.
>>>> Here is the error log:
>>>>
>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in
>>>> ../../../petsc/src/sys/objects/pinit.c
>>>> [1]PETSC ERROR: #2 checkError() line 56 in
>>>> ../../../physics/src/eigensolver/SLEPc.cpp
>>>> [1]PETSC ERROR: #3 PetscInitialize() line 966 in
>>>> ../../../petsc/src/sys/objects/pinit.c
>>>> [1]PETSC ERROR: #4 SlepcInitialize() line 262 in
>>>> ../../../slepc/src/sys/slepcinit.c
>>>> [0]PETSC ERROR: #1 PetscInitialize() line 969 in
>>>> ../../../petsc/src/sys/objects/pinit.c
>>>> [0]PETSC ERROR: #2 checkError() line 56 in
>>>> ../../../physics/src/eigensolver/SLEPc.cpp
>>>> [0]PETSC ERROR: #3 PetscInitialize() line 966 in
>>>> ../../../petsc/src/sys/objects/pinit.c
>>>> [0]PETSC ERROR: #4 SlepcInitialize() line 262 in
>>>> ../../../slepc/src/sys/slepcinit.c
>>>> PETSC ERROR: Logging has not been enabled.
>>>> You might have forgotten to call PetscInitialize().
>>>> PETSC ERROR: Logging has not been enabled.
>>>> You might have forgotten to call PetscInitialize().
>>>>
>>>> --------------------------------------------------------------------------
>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>>> with errorcode 56.
>>>>
>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>>> You may or may not see output from other processes, depending on
>>>> exactly when Open MPI kills them.
>>>>
>>>> Thanks,
>>>> Sam
>>>>
>>>> On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang <junchao.zhang at gmail.com>
>>>> wrote:
>>>>
>>>>> Sam,
>>>>> The MPI_Comm_create_keyval() error was fixed in maint/master. From
>>>>> the error message, it seems you need to configure --with-log=1
>>>>> Otherwise, please send your full error stack trace and
>>>>> configure.log.
>>>>> Thanks.
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Thu, Jun 25, 2020 at 2:18 PM Sam Guo <sam.guo at cd-adapco.com> wrote:
>>>>>
>>>>>> Hi Junchao,
>>>>>> I now encountered the same error with parallel. I am wondering if
>>>>>> there is a need for parallel fix as well.
>>>>>> [1]PETSC ERROR: #1 PetscInitialize() line 969 in
>>>>>> ../../../petsc/src/sys/objects/pinit.c
>>>>>> PETSC ERROR: Logging has not been enabled.
>>>>>> You might have forgotten to call PetscInitialize().
>>>>>> PETSC ERROR: Logging has not been enabled.
>>>>>> You might have forgotten to call PetscInitialize().
>>>>>>
>>>>>> On Sat, Jun 20, 2020 at 7:35 PM Sam Guo <sam.guo at cd-adapco.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Junchao,
>>>>>>> Your patch works.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sam
>>>>>>>
>>>>>>> On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang <
>>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Jun 20, 2020 at 12:24 PM Barry Smith <bsmith at petsc.dev>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Junchao,
>>>>>>>>>
>>>>>>>>> This is a good bug fix. It solves the problem when PETSc
>>>>>>>>> initialize is called many times.
>>>>>>>>>
>>>>>>>>> There is another fix you can do to limit PETSc mpiuni running
>>>>>>>>> out of attributes inside a single PETSc run:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> int MPI_Comm_create_keyval(MPI_Copy_function
>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)
>>>>>>>>> {
>>>>>>>>>
>>>>>>>>> if (num_attr >= MAX_ATTR){
>>>>>>>>> for (i=0; i<num_attr; i++) {
>>>>>>>>> if (!attr_keyval[i].extra_state) {
>>>>>>>>>
>>>>>>>> attr_keyval[i].extra_state is provided by user (could be NULL). We
>>>>>>>> can not rely on it.
>>>>>>>>
>>>>>>>>> /* reuse this slot */
>>>>>>>>> attr_keyval[i].extra_state = extra_state;
>>>>>>>>> attr_keyval[i.]del = delete_fn;
>>>>>>>>> *keyval = i;
>>>>>>>>> return MPI_SUCCESS;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1);
>>>>>>>>> }
>>>>>>>>> return MPIUni_Abort(MPI_COMM_WORLD,1);
>>>>>>>>> attr_keyval[num_attr].extra_state = extra_state;
>>>>>>>>> attr_keyval[num_attr].del = delete_fn;
>>>>>>>>> *keyval = num_attr++;
>>>>>>>>> return MPI_SUCCESS;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> This will work if the user creates tons of attributes but is
>>>>>>>>> constantly deleting some as they new ones. So long as the number
>>>>>>>>> outstanding at one time is < MAX_ATTR)
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang <
>>>>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I don't understand what your session means. Let's try this patch
>>>>>>>>>
>>>>>>>>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c
>>>>>>>>> index d559a513..c058265d 100644
>>>>>>>>> --- a/src/sys/mpiuni/mpi.c
>>>>>>>>> +++ b/src/sys/mpiuni/mpi.c
>>>>>>>>> @@ -283,6 +283,7 @@ int MPI_Finalize(void)
>>>>>>>>> MPI_Comm_free(&comm);
>>>>>>>>> comm = MPI_COMM_SELF;
>>>>>>>>> MPI_Comm_free(&comm);
>>>>>>>>> + num_attr = 1; /* reset the counter */
>>>>>>>>> MPI_was_finalized = 1;
>>>>>>>>> return MPI_SUCCESS;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --Junchao Zhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo <sam.guo at cd-adapco.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Typo: I mean “Assuming initializer is only needed once for entire
>>>>>>>>>> session”
>>>>>>>>>>
>>>>>>>>>> On Saturday, June 20, 2020, Sam Guo <sam.guo at cd-adapco.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Assuming finalizer is only needed once for entire session(?), I
>>>>>>>>>>> can put initializer into the static block to call it once but where do I
>>>>>>>>>>> call finalizer?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Saturday, June 20, 2020, Junchao Zhang <
>>>>>>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The counter num_attr should be recycled. But first try to call
>>>>>>>>>>>> PETSc initialize/Finalize only once to see it fixes the error.
>>>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo <sam.guo at cd-adapco.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> To clarify, I call PETSc initialize and PETSc finalize
>>>>>>>>>>>>> everytime I call SLEPc:
>>>>>>>>>>>>>
>>>>>>>>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr);
>>>>>>>>>>>>>
>>>>>>>>>>>>> SlepcInitialize(&argc,&args,static_cast<char*>(nullptr),help);
>>>>>>>>>>>>>
>>>>>>>>>>>>> //calling slepc
>>>>>>>>>>>>>
>>>>>>>>>>>>> SlepcFinalize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> PetscFinalize();
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo <
>>>>>>>>>>>>> sam.guo at cd-adapco.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear PETSc team,
>>>>>>>>>>>>>> When I called SLEPc multiple time, I eventually got
>>>>>>>>>>>>>> following error:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers
>>>>>>>>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in
>>>>>>>>>>>>>> ../../../petsc/src/sys/objects/pinit.c
>>>>>>>>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in
>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c
>>>>>>>>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in
>>>>>>>>>>>>>> ../../../slepc/src/sys/slepcinit.c
>>>>>>>>>>>>>> PETSC ERROR: Logging has not been enabled.
>>>>>>>>>>>>>> You might have forgotten to call PetscInitialize().
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I debugged: it is because of following in
>>>>>>>>>>>>>> petsc/src/sys/mpiuni/mpi.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (num_attr >= MAX_ATTR)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function
>>>>>>>>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> num_attr is declared static and keeps increasing every
>>>>>>>>>>>>>> time MPI_Comm_create_keyval is called.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is this a bug or I didn't use it correctly?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Sam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200626/c0e15800/attachment-0001.html>
More information about the petsc-users
mailing list