[petsc-users] KSP_Solve crashes in debug mode

[Barry Smith]

  Hmm, it is crashing in PetscFunctionListDLAllPop_Private() which is very recent code with Jacob's name on it.

  It is only run in debug mode, hence only crashes there. I don't know why it fails here, somehow memory was corrupted earlier? Try running with -malloc_debug or under valgrind?

static PetscErrorCode PetscFunctionListDLAllPop_Private(PetscFunctionList fl)
{
  PetscFunctionBegin;
  if (PetscDefined(USE_DEBUG) && !PetscDefined(HAVE_THREADSAFETY)) {
    PetscFunctionListDLAll current = dlallhead, prev = NULL;

    /* Remove this entry from the main DL list (if it is in it) */
    while (current) {
      const PetscFunctionListDLAll next = current->next;

      if (current->data == fl) {
        if (prev) {
          // somewhere in the middle (or end) of the list
          prev->next = next;
        } else {
          // prev = NULL implies current = dlallhead, so front of list
          dlallhead = next;
        }
        PetscCall(PetscFree(current));
        break;
      }
      prev    = current;
      current = next;
    }
  }
  PetscFunctionReturn(PETSC_SUCCESS);
}



> On Feb 9, 2023, at 1:56 PM, Sajid Ali Syed <sasyed at fnal.gov> wrote:
> 
> Hi Barry, 
> 
> The lack of line numbers is due to the fact that this build of PETSc was done via spack which installs it in a temporary directory before moving it to the final location. 
> 
> I have removed that build and installed PETSc locally (albeit with a simpler configuration) and see the same bug. Logs for this configuration and the error trace with this build are attached with this email. 
> 
> Thank You,
> Sajid Ali (he/him) | Research Associate
> Scientific Computing Division
> Fermi National Accelerator Laboratory
> s-sajid-ali.github.io <http://s-sajid-ali.github.io/>
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: Thursday, February 9, 2023 12:02 PM
> To: Sajid Ali Syed <sasyed at fnal.gov <mailto:sasyed at fnal.gov>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode
>  
> 
>   Hmm, looks like your build may be funny? It is not in debug mode 
> 
> frame #2: 0x000000010eda20c8 libpetsc.3.018.dylib`PetscHeaderDestroy_Private + 1436
>     frame #3: 0x000000010f10176c libpetsc.3.018.dylib`VecDestroy + 808
>     frame #4: 0x0000000110199f34 libpetsc.3.018.dylib`KSPSolve_Private + 512
> 
> In debugger mode it would show the line numbers where the crash occurred and help us determine the problem. I do note the -g being used by the compilers so cannot explain off hand why it does not display the debug information.
> 
>   Barry
> 
> 
>> On Feb 9, 2023, at 12:42 PM, Sajid Ali Syed via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>> 
>> Hi PETSc-developers, 
>> 
>> In our application we call KSP_Solve as part of a step to propagate a beam through a lattice. I am observing a crash within KSP_Solve for an application only after the 43rd call to KSP_Solve when building the application and PETSc in debug mode, full logs for which are attached with this email (1 MPI rank and 4 OMP threads were used, but this crash occurs with multiple MPI ranks as well ). I am also including the last few lines of the configuration for this build. This crash does not occur when building the application and PETSc in release mode. 
>> 
>> Could someone tell me what causes this crash and if anything can be done to prevent it? Thanks in advance.
>> 
>> The configuration of this solver is here : 
>> 
>> https://github.com/fnalacceleratormodeling/synergia2/blob/sajid/features/openpmd_basic_integration/src/synergia/collective/space_charge_3d_fd_utils.cc#L273-L292 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fnalacceleratormodeling_synergia2_blob_sajid_features_openpmd-5Fbasic-5Fintegration_src_synergia_collective_space-5Fcharge-5F3d-5Ffd-5Futils.cc-23L273-2DL292&d=DwMFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=fZRIP-srEU-rOVw461XeAmMY4_3VyZ1mujQae_3pw33K9hMFg-5yyo_sXsQJ_1dn&s=YToKoRAfim51m6aN-z4FlzfM0UDy46D7QJh8B11shTg&e=>
>> 
>> Thank You,
>> Sajid Ali (he/him) | Research Associate
>> Scientific Computing Division
>> Fermi National Accelerator Laboratory
>> s-sajid-ali.github.io <https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io_&d=DwMFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=fZRIP-srEU-rOVw461XeAmMY4_3VyZ1mujQae_3pw33K9hMFg-5yyo_sXsQJ_1dn&s=g4Ch_bfIjw1n_m0d0zSXYc_eDI-4eJc9zSA_RfhUFOQ&e=>
>> <ksp_crash_log>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230209/796af1d6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure_log_tail_local_install
Type: application/octet-stream
Size: 2781 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230209/796af1d6/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230209/796af1d6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ksp_crash_log_local_install
Type: application/octet-stream
Size: 5179 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230209/796af1d6/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230209/796af1d6/attachment-0002.html>