[petsc-users] MPI barrier issue using MatZeroRows
Matthew Knepley
knepley at gmail.com
Wed Nov 29 19:14:39 CST 2023
On Wed, Nov 29, 2023 at 7:27 PM Amneet Bhalla <mail2amneet at gmail.com> wrote:
> Ah, I also tried without step 2 (i.e., manually doing MPI_allgatherv for
> Dirichlet rows), and that also works. So it seems that each processor needs
> to send in their own Dirichlet rows, and not a union of them. Is that
> correct?
>
Yes, that is correct.
Thanks,
Matt
> On Wed, Nov 29, 2023 at 3:48 PM Amneet Bhalla <mail2amneet at gmail.com>
> wrote:
>
>> Thanks Barry! I tried that and it seems to be working. This is what I
>> did. It would be great if you could take a look at it and let me know if
>> this is what you had in mind.
>>
>> 1. Collected Dirichlet rows locally
>>
>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L731
>>
>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L797
>>
>>
>> 2. MPI_allgatherv Dirichlet rows
>>
>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L805-L810
>>
>> 3. Called the MatZeroRows function
>>
>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L812-L814
>>
>>
>>
>>
>> On Wed, Nov 29, 2023 at 11:32 AM Barry Smith <bsmith at petsc.dev> wrote:
>>
>>>
>>>
>>> On Nov 29, 2023, at 2:11 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Wed, Nov 29, 2023 at 1:55 PM Amneet Bhalla <mail2amneet at gmail.com>
>>> wrote:
>>>
>>>> So the code logic is after the matrix is assembled, I iterate over all
>>>> distributed patches in the domain to see which of the patch is abutting a
>>>> Dirichlet boundary. Depending upon which patch abuts a physical and
>>>> Dirichlet boundary, a processor will call this routine. However, that same
>>>> processor is “owning” that DoF, which would be on its diagonal.
>>>>
>>>> I think Barry already mentioned this is not going to work unless I use
>>>> the flag to not communicate explicitly. However, that flag is not working
>>>> as it should over here for some reason.
>>>>
>>>
>>> Oh, I do not think that is right.
>>>
>>> Barry, when I look at the code, MPIU_Allreduce is always going to be
>>> called to fix up the nonzero_state. Am I wrong about that?
>>>
>>>
>>> No, you are correct. I missed that in my earlier look. Setting those
>>> flags reduce the number of MPI reductions but does not eliminate them
>>> completely.
>>>
>>> MatZeroRows is collective (as its manual page indicates) so you have
>>> to do the second thing I suggested. Inside your for loop construct an array
>>> containing all the local
>>> rows being zeroed and then make a single call by all MPI processes to
>>> MatZeroRows(). Note this is a small change of just a handful of lines of
>>> code.
>>>
>>> Barry
>>>
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> I can always change the matrix coefficients for Dirichlet rows during
>>>> MatSetValues. However, that would lengthen my code and I was trying to
>>>> avoid that.
>>>>
>>>> On Wed, Nov 29, 2023 at 10:02 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Nov 29, 2023 at 12:30 PM Amneet Bhalla <mail2amneet at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ok, I added both, but it still hangs. Here, is bt from all three
>>>>>> tasks:
>>>>>>
>>>>>
>>>>> It looks like two processes are calling AllReduce, but one is not. Are
>>>>> all procs not calling MatZeroRows?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Task 1:
>>>>>>
>>>>>> amneetb at APSB-MacBook-Pro-16:~$ lldb -p 44691
>>>>>> (lldb) process attach --pid 44691
>>>>>> Process 44691 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000018a2d750c
>>>>>> libsystem_kernel.dylib`__semwait_signal + 8
>>>>>> libsystem_kernel.dylib`:
>>>>>> -> 0x18a2d750c <+8>: b.lo 0x18a2d752c ; <+40>
>>>>>> 0x18a2d7510 <+12>: pacibsp
>>>>>> 0x18a2d7514 <+16>: stp x29, x30, [sp, #-0x10]!
>>>>>> 0x18a2d7518 <+20>: mov x29, sp
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> Executable module set to
>>>>>> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
>>>>>> Architecture set to: arm64-apple-macosx-.
>>>>>> (lldb) cont
>>>>>> Process 44691 resuming
>>>>>> Process 44691 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000010ba40b60
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_release + 752
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_release:
>>>>>> -> 0x10ba40b60 <+752>: add w8, w8, #0x1
>>>>>> 0x10ba40b64 <+756>: ldr w9, [x22]
>>>>>> 0x10ba40b68 <+760>: cmp w8, w9
>>>>>> 0x10ba40b6c <+764>: b.lt 0x10ba40b4c ; <+732>
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> (lldb) bt
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> * frame #0: 0x000000010ba40b60
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_release + 752
>>>>>> frame #1: 0x000000010ba48528
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_allreduce_release_gather + 1088
>>>>>> frame #2: 0x000000010ba47964
>>>>>> libpmpi.12.dylib`MPIDI_Allreduce_intra_composition_gamma + 368
>>>>>> frame #3: 0x000000010ba35e78 libpmpi.12.dylib`MPIR_Allreduce +
>>>>>> 1588
>>>>>> frame #4: 0x0000000103f587dc libmpi.12.dylib`MPI_Allreduce + 2280
>>>>>> frame #5: 0x0000000106d67650
>>>>>> libpetsc.3.17.dylib`MatZeroRows_MPIAIJ(A=0x0000000105846470, N=1,
>>>>>> rows=0x000000016dbfa9f4, diag=1, x=0x0000000000000000,
>>>>>> b=0x0000000000000000) at mpiaij.c:827:3
>>>>>> frame #6: 0x0000000106aadfac
>>>>>> libpetsc.3.17.dylib`MatZeroRows(mat=0x0000000105846470, numRows=1,
>>>>>> rows=0x000000016dbfa9f4, diag=1, x=0x0000000000000000,
>>>>>> b=0x0000000000000000) at matrix.c:5935:3
>>>>>> frame #7: 0x00000001023952d0
>>>>>> fo_acoustic_streaming_solver_2d`IBAMR::AcousticStreamingPETScMatUtilities::constructPatchLevelFOAcousticStreamingOp(mat=0x000000016dc04168,
>>>>>> omega=1, sound_speed=1, rho_idx=3, mu_idx=2, lambda_idx=4,
>>>>>> u_bc_coefs=0x000000016dc04398, data_time=NaN, num_dofs_per_proc=size=3,
>>>>>> u_dof_index_idx=27, p_dof_index_idx=28,
>>>>>> patch_level=Pointer<SAMRAI::hier::PatchLevel<2> > @ 0x000000016dbfcec0,
>>>>>> mu_interp_type=VC_HARMONIC_INTERP) at
>>>>>> AcousticStreamingPETScMatUtilities.cpp:799:36
>>>>>> frame #8: 0x00000001023acb8c
>>>>>> fo_acoustic_streaming_solver_2d`IBAMR::FOAcousticStreamingPETScLevelSolver::initializeSolverStateSpecialized(this=0x000000016dc04018,
>>>>>> x=0x000000016dc05778, (null)=0x000000016dc05680) at
>>>>>> FOAcousticStreamingPETScLevelSolver.cpp:149:5
>>>>>> frame #9: 0x000000010254a2dc
>>>>>> fo_acoustic_streaming_solver_2d`IBTK::PETScLevelSolver::initializeSolverState(this=0x000000016dc04018,
>>>>>> x=0x000000016dc05778, b=0x000000016dc05680) at PETScLevelSolver.cpp:
>>>>>> 340:5
>>>>>> frame #10: 0x0000000102202e5c
>>>>>> fo_acoustic_streaming_solver_2d`main(argc=11, argv=0x000000016dc07450) at
>>>>>> fo_acoustic_streaming_solver.cpp:400:22
>>>>>> frame #11: 0x0000000189fbbf28 dyld`start + 2236
>>>>>> (lldb)
>>>>>>
>>>>>>
>>>>>> Task 2:
>>>>>>
>>>>>> amneetb at APSB-MacBook-Pro-16:~$ lldb -p 44692
>>>>>> (lldb) process attach --pid 44692
>>>>>> Process 44692 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000018a2d750c
>>>>>> libsystem_kernel.dylib`__semwait_signal + 8
>>>>>> libsystem_kernel.dylib`:
>>>>>> -> 0x18a2d750c <+8>: b.lo 0x18a2d752c ; <+40>
>>>>>> 0x18a2d7510 <+12>: pacibsp
>>>>>> 0x18a2d7514 <+16>: stp x29, x30, [sp, #-0x10]!
>>>>>> 0x18a2d7518 <+20>: mov x29, sp
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> Executable module set to
>>>>>> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
>>>>>> Architecture set to: arm64-apple-macosx-.
>>>>>> (lldb) cont
>>>>>> Process 44692 resuming
>>>>>> Process 44692 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000010e5a022c
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 516
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:
>>>>>> -> 0x10e5a022c <+516>: ldr x10, [x19, #0x4e8]
>>>>>> 0x10e5a0230 <+520>: cmp x9, x10
>>>>>> 0x10e5a0234 <+524>: b.hs 0x10e5a0254 ; <+556>
>>>>>> 0x10e5a0238 <+528>: add w8, w8, #0x1
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> (lldb) bt
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> * frame #0: 0x000000010e5a022c
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 516
>>>>>> frame #1: 0x000000010e59fd14
>>>>>> libpmpi.12.dylib`MPIDI_SHM_mpi_barrier + 224
>>>>>> frame #2: 0x000000010e59fb60
>>>>>> libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44
>>>>>> frame #3: 0x000000010e585490 libpmpi.12.dylib`MPIR_Barrier + 900
>>>>>> frame #4: 0x0000000106ac5030 libmpi.12.dylib`MPI_Barrier + 684
>>>>>> frame #5: 0x0000000108e62638
>>>>>> libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=1140850688,
>>>>>> comm_out=0x00000001408ae4b0, first_tag=0x00000001408ae4e4) at tagm.c:
>>>>>> 235:5
>>>>>> frame #6: 0x0000000108e6a910
>>>>>> libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x00000001408ae470,
>>>>>> classid=1211228, class_name="KSP", descr="Krylov Method", mansec="KSP",
>>>>>> comm=1140850688, destroy=(libpetsc.3.17.dylib`KSPDestroy at itfunc.c:1418),
>>>>>> view=(libpetsc.3.17.dylib`KSPView at itcreate.c:113)) at inherit.c:62
>>>>>> :3
>>>>>> frame #7: 0x000000010aa28010
>>>>>> libpetsc.3.17.dylib`KSPCreate(comm=1140850688, inksp=0x000000016b0a4160) at
>>>>>> itcreate.c:679:3
>>>>>> frame #8: 0x00000001050aa2f4
>>>>>> fo_acoustic_streaming_solver_2d`IBTK::PETScLevelSolver::initializeSolverState(this=0x000000016b0a4018,
>>>>>> x=0x000000016b0a5778, b=0x000000016b0a5680) at PETScLevelSolver.cpp:
>>>>>> 344:12
>>>>>> frame #9: 0x0000000104d62e5c
>>>>>> fo_acoustic_streaming_solver_2d`main(argc=11, argv=0x000000016b0a7450) at
>>>>>> fo_acoustic_streaming_solver.cpp:400:22
>>>>>> frame #10: 0x0000000189fbbf28 dyld`start + 2236
>>>>>> (lldb)
>>>>>>
>>>>>>
>>>>>> Task 3:
>>>>>>
>>>>>> amneetb at APSB-MacBook-Pro-16:~$ lldb -p 44693
>>>>>> (lldb) process attach --pid 44693
>>>>>> Process 44693 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000018a2d750c
>>>>>> libsystem_kernel.dylib`__semwait_signal + 8
>>>>>> libsystem_kernel.dylib`:
>>>>>> -> 0x18a2d750c <+8>: b.lo 0x18a2d752c ; <+40>
>>>>>> 0x18a2d7510 <+12>: pacibsp
>>>>>> 0x18a2d7514 <+16>: stp x29, x30, [sp, #-0x10]!
>>>>>> 0x18a2d7518 <+20>: mov x29, sp
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> Executable module set to
>>>>>> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
>>>>>> Architecture set to: arm64-apple-macosx-.
>>>>>> (lldb) cont
>>>>>> Process 44693 resuming
>>>>>> Process 44693 stopped
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> frame #0: 0x000000010e59c68c
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_gather + 952
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_gather:
>>>>>> -> 0x10e59c68c <+952>: ldr w9, [x21]
>>>>>> 0x10e59c690 <+956>: cmp w8, w9
>>>>>> 0x10e59c694 <+960>: b.lt 0x10e59c670 ; <+924>
>>>>>> 0x10e59c698 <+964>: bl 0x10e59ce64 ;
>>>>>> MPID_Progress_test
>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>> (lldb) bt
>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>> SIGSTOP
>>>>>> * frame #0: 0x000000010e59c68c
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_release_gather_gather + 952
>>>>>> frame #1: 0x000000010e5a44bc
>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_allreduce_release_gather + 980
>>>>>> frame #2: 0x000000010e5a3964
>>>>>> libpmpi.12.dylib`MPIDI_Allreduce_intra_composition_gamma + 368
>>>>>> frame #3: 0x000000010e591e78 libpmpi.12.dylib`MPIR_Allreduce +
>>>>>> 1588
>>>>>> frame #4: 0x0000000106ab47dc libmpi.12.dylib`MPI_Allreduce + 2280
>>>>>> frame #5: 0x00000001098c3650
>>>>>> libpetsc.3.17.dylib`MatZeroRows_MPIAIJ(A=0x0000000136862270, N=1,
>>>>>> rows=0x000000016b09e9f4, diag=1, x=0x0000000000000000,
>>>>>> b=0x0000000000000000) at mpiaij.c:827:3
>>>>>> frame #6: 0x0000000109609fac
>>>>>> libpetsc.3.17.dylib`MatZeroRows(mat=0x0000000136862270, numRows=1,
>>>>>> rows=0x000000016b09e9f4, diag=1, x=0x0000000000000000,
>>>>>> b=0x0000000000000000) at matrix.c:5935:3
>>>>>> frame #7: 0x0000000104ef12d0
>>>>>> fo_acoustic_streaming_solver_2d`IBAMR::AcousticStreamingPETScMatUtilities::constructPatchLevelFOAcousticStreamingOp(mat=0x000000016b0a8168,
>>>>>> omega=1, sound_speed=1, rho_idx=3, mu_idx=2, lambda_idx=4,
>>>>>> u_bc_coefs=0x000000016b0a8398, data_time=NaN, num_dofs_per_proc=size=3,
>>>>>> u_dof_index_idx=27, p_dof_index_idx=28,
>>>>>> patch_level=Pointer<SAMRAI::hier::PatchLevel<2> > @ 0x000000016b0a0ec0,
>>>>>> mu_interp_type=VC_HARMONIC_INTERP) at
>>>>>> AcousticStreamingPETScMatUtilities.cpp:799:36
>>>>>> frame #8: 0x0000000104f08b8c
>>>>>> fo_acoustic_streaming_solver_2d`IBAMR::FOAcousticStreamingPETScLevelSolver::initializeSolverStateSpecialized(this=0x000000016b0a8018,
>>>>>> x=0x000000016b0a9778, (null)=0x000000016b0a9680) at
>>>>>> FOAcousticStreamingPETScLevelSolver.cpp:149:5
>>>>>> frame #9: 0x00000001050a62dc
>>>>>> fo_acoustic_streaming_solver_2d`IBTK::PETScLevelSolver::initializeSolverState(this=0x000000016b0a8018,
>>>>>> x=0x000000016b0a9778, b=0x000000016b0a9680) at PETScLevelSolver.cpp:
>>>>>> 340:5
>>>>>> frame #10: 0x0000000104d5ee5c
>>>>>> fo_acoustic_streaming_solver_2d`main(argc=11, argv=0x000000016b0ab450) at
>>>>>> fo_acoustic_streaming_solver.cpp:400:22
>>>>>> frame #11: 0x0000000189fbbf28 dyld`start + 2236
>>>>>> (lldb)
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 29, 2023 at 7:22 AM Barry Smith <bsmith at petsc.dev> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Nov 29, 2023, at 1:16 AM, Amneet Bhalla <mail2amneet at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> BTW, I think you meant using MatSetOption(mat,
>>>>>>> *MAT_NO_OFF_PROC_ZERO_ROWS*, PETSC_TRUE)
>>>>>>>
>>>>>>>
>>>>>>> Yes
>>>>>>>
>>>>>>> instead ofMatSetOption(mat, *MAT_NO_OFF_PROC_ENTRIES*, PETSC_TRUE)
>>>>>>> ??
>>>>>>>
>>>>>>>
>>>>>>> Please try setting both flags.
>>>>>>>
>>>>>>> However, that also did not help to overcome the MPI Barrier issue.
>>>>>>>
>>>>>>>
>>>>>>> If there is still a problem please trap all the MPI processes when
>>>>>>> they hang in the debugger and send the output from using bt on all of them.
>>>>>>> This way
>>>>>>> we can see the different places the different MPI processes are
>>>>>>> stuck at.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 28, 2023 at 9:57 PM Amneet Bhalla <mail2amneet at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I added that option but the code still gets stuck at the same call
>>>>>>>> MatZeroRows with 3 processors.
>>>>>>>>
>>>>>>>> On Tue, Nov 28, 2023 at 7:23 PM Amneet Bhalla <
>>>>>>>> mail2amneet at gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 28, 2023 at 6:42 PM Barry Smith <bsmith at petsc.dev>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> for (int comp = 0; comp < 2; ++comp)
>>>>>>>>>> {
>>>>>>>>>> .......
>>>>>>>>>> for (Box<NDIM>::Iterator bc(bc_coef_box); bc;
>>>>>>>>>> bc++)
>>>>>>>>>> {
>>>>>>>>>> ......
>>>>>>>>>> if (IBTK::abs_equal_eps(b, 0.0))
>>>>>>>>>> {
>>>>>>>>>> const double diag_value = a;
>>>>>>>>>> ierr = MatZeroRows(mat, 1,
>>>>>>>>>> &u_dof_index, diag_value, NULL, NULL);
>>>>>>>>>> IBTK_CHKERRQ(ierr);
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> In general, this code will not work because each process calls
>>>>>>>>>> MatZeroRows a different number of times, so it cannot match up with all the
>>>>>>>>>> processes.
>>>>>>>>>>
>>>>>>>>>> If u_dof_index is always local to the current process, you can
>>>>>>>>>> call MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE) above the for
>>>>>>>>>> loop and
>>>>>>>>>> the MatZeroRows will not synchronize across the MPI processes
>>>>>>>>>> (since it does not need to and you told it that).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, u_dof_index is going to be local and I put a check on it a
>>>>>>>>> few lines before calling MatZeroRows.
>>>>>>>>>
>>>>>>>>> Can MatSetOption() be called after the matrix has been assembled?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> If the u_dof_index will not always be local, then you need, on
>>>>>>>>>> each process, to list all the u_dof_index for each process in an array and
>>>>>>>>>> then call MatZeroRows()
>>>>>>>>>> once after the loop so it can exchange the needed information
>>>>>>>>>> with the other MPI processes to get the row indices to the right place.
>>>>>>>>>>
>>>>>>>>>> Barry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Nov 28, 2023, at 6:44 PM, Amneet Bhalla <mail2amneet at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Folks,
>>>>>>>>>>
>>>>>>>>>> I am using MatZeroRows() to set Dirichlet boundary conditions.
>>>>>>>>>> This works fine for the serial run and the solver produces correct results
>>>>>>>>>> (verified through analytical solution). However, when I run the case in
>>>>>>>>>> parallel, the simulation gets stuck at MatZeroRows(). My understanding is
>>>>>>>>>> that this function needs to be called after the MatAssemblyBegin{End}() has
>>>>>>>>>> been called, and should be called by all processors. Here is that bit of
>>>>>>>>>> the code which calls MatZeroRows() after the matrix has been assembled
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L724-L801
>>>>>>>>>>
>>>>>>>>>> I ran the parallel code (on 3 processors) in the debugger
>>>>>>>>>> (-start_in_debugger). Below is the call stack from the processor that gets
>>>>>>>>>> stuck
>>>>>>>>>>
>>>>>>>>>> amneetb at APSB-MBP-16:~$ lldb -p 4307
>>>>>>>>>> (lldb) process attach --pid 4307
>>>>>>>>>> Process 4307 stopped
>>>>>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>>>>>> SIGSTOP
>>>>>>>>>> frame #0: 0x000000018a2d750c
>>>>>>>>>> libsystem_kernel.dylib`__semwait_signal + 8
>>>>>>>>>> libsystem_kernel.dylib`:
>>>>>>>>>> -> 0x18a2d750c <+8>: b.lo 0x18a2d752c ; <+40>
>>>>>>>>>> 0x18a2d7510 <+12>: pacibsp
>>>>>>>>>> 0x18a2d7514 <+16>: stp x29, x30, [sp, #-0x10]!
>>>>>>>>>> 0x18a2d7518 <+20>: mov x29, sp
>>>>>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>>>>>> Executable module set to
>>>>>>>>>> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
>>>>>>>>>> Architecture set to: arm64-apple-macosx-.
>>>>>>>>>> (lldb) cont
>>>>>>>>>> Process 4307 resuming
>>>>>>>>>> Process 4307 stopped
>>>>>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>>>>>> SIGSTOP
>>>>>>>>>> frame #0: 0x0000000109d281b8
>>>>>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
>>>>>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:
>>>>>>>>>> -> 0x109d281b8 <+400>: ldr w9, [x24]
>>>>>>>>>> 0x109d281bc <+404>: cmp w8, w9
>>>>>>>>>> 0x109d281c0 <+408>: b.lt 0x109d281a0 ; <+376>
>>>>>>>>>> 0x109d281c4 <+412>: bl 0x109d28e64 ;
>>>>>>>>>> MPID_Progress_test
>>>>>>>>>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>>>>>>>>>> (lldb) bt
>>>>>>>>>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>>>>>>>>>> SIGSTOP
>>>>>>>>>> * frame #0: 0x0000000109d281b8
>>>>>>>>>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
>>>>>>>>>> frame #1: 0x0000000109d27d14
>>>>>>>>>> libpmpi.12.dylib`MPIDI_SHM_mpi_barrier + 224
>>>>>>>>>> frame #2: 0x0000000109d27b60
>>>>>>>>>> libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44
>>>>>>>>>> frame #3: 0x0000000109d0d490 libpmpi.12.dylib`MPIR_Barrier +
>>>>>>>>>> 900
>>>>>>>>>> frame #4: 0x000000010224d030 libmpi.12.dylib`MPI_Barrier +
>>>>>>>>>> 684
>>>>>>>>>> frame #5: 0x00000001045ea638
>>>>>>>>>> libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=-2080374782,
>>>>>>>>>> comm_out=0x000000010300bcb0, first_tag=0x000000010300bce4) at
>>>>>>>>>> tagm.c:235:5
>>>>>>>>>> frame #6: 0x00000001045f2910
>>>>>>>>>> libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x000000010300bc70,
>>>>>>>>>> classid=1211227, class_name="PetscSF", descr="Star Forest",
>>>>>>>>>> mansec="PetscSF", comm=-2080374782,
>>>>>>>>>> destroy=(libpetsc.3.17.dylib`PetscSFDestroy at sf.c:224),
>>>>>>>>>> view=(libpetsc.3.17.dylib`PetscSFView at sf.c:841)) at inherit.c:
>>>>>>>>>> 62:3
>>>>>>>>>> frame #7: 0x00000001049cf820
>>>>>>>>>> libpetsc.3.17.dylib`PetscSFCreate(comm=-2080374782, sf=0x000000016f911a50)
>>>>>>>>>> at sf.c:62:3
>>>>>>>>>> frame #8: 0x0000000104cd3024
>>>>>>>>>> libpetsc.3.17.dylib`MatZeroRowsMapLocal_Private(A=0x00000001170c1270, N=1,
>>>>>>>>>> rows=0x000000016f912cb4, nr=0x000000016f911df8, olrows=0x000000016f911e00)
>>>>>>>>>> at zerorows.c:36:5
>>>>>>>>>> frame #9: 0x000000010504ea50
>>>>>>>>>> libpetsc.3.17.dylib`MatZeroRows_MPIAIJ(A=0x00000001170c1270, N=1,
>>>>>>>>>> rows=0x000000016f912cb4, diag=1, x=0x0000000000000000,
>>>>>>>>>> b=0x0000000000000000) at mpiaij.c:768:3
>>>>>>>>>>
>>>>>>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>>
>>>
>>
>> --
>> --Amneet
>>
>>
>>
>>
>
> --
> --Amneet
>
>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231129/cfcca990/attachment-0001.html>
More information about the petsc-users
mailing list