[petsc-users] MPI error for large number of processes and subcomms
Randall Mackie
rlmackie862 at gmail.com
Thu Apr 30 19:46:34 CDT 2020
Hi Junchao,
Unfortunately these modifications did not work on our cluster (see output below).
However, I am not asking you to spend anymore time on this, as we are able to avoid the problem by setting appropriate sysctl parameters into /etc/sysctl.conf.
Thank you again for all your help on this.
Randy
Output of test program:
mpiexec -np 1280 -hostfile machines ./test -nsubs 160 -nx 100 -ny 100 -nz 10 -max_pending_isends 64
Started
ind2 max 31999999
nis 33600
begin VecScatter create
[1175]PETSC ERROR: #1 PetscCommBuildTwoSided_Ibarrier() line 102 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/sys/utils/mpits.c
[1175]PETSC ERROR: #2 PetscCommBuildTwoSided() line 313 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/sys/utils/mpits.c
[1175]PETSC ERROR: #3 PetscSFSetUp_Basic() line 33 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/vec/is/sf/impls/basic/sfbasic.c
[1175]PETSC ERROR: #4 PetscSFSetUp() line 253 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/vec/is/sf/interface/sf.c
[1175]PETSC ERROR: #5 VecScatterSetUp_SF() line 747 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/vec/vscat/impls/sf/vscatsf.c
[1175]PETSC ERROR: #6 VecScatterSetUp() line 208 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/vec/vscat/interface/vscatfce.c
[1175]PETSC ERROR: #7 VecScatterCreate() line 287 in /state/std2/FEMI/PETSc/petsc-jczhang-throttle-pending-isends/src/vec/vscat/interface/vscreate.c
> On Apr 27, 2020, at 9:59 AM, Junchao Zhang <junchao.zhang at gmail.com> wrote:
>
> Randy,
> You are absolutely right. The AOApplicationToPetsc could not be removed. Since the excessive communication is inevitable, I made two changes in petsc to ease that. One is I skewed the communication to let each rank send to ranks greater than itself first. The other is an option, -max_pending_isend, to control number of pending isends. Current default is 512.
> I have an MR at https://gitlab.com/petsc/petsc/-/merge_requests/2757 <https://gitlab.com/petsc/petsc/-/merge_requests/2757>. I tested it dozens of times with your example at 5120 ranks. It worked fine.
> Please try it in your environment and let me know the result. Since the failure is random, you may need to run multiple times.
>
> BTW, if no objection, I'd like to add your excellent example to petsc repo.
>
> Thanks
> --Junchao Zhang
>
>
> On Fri, Apr 24, 2020 at 5:32 PM Randall Mackie <rlmackie862 at gmail.com <mailto:rlmackie862 at gmail.com>> wrote:
> Hi Junchao,
>
> I tested by commenting out the AOApplicationToPetsc calls as you suggest, but it doesn’t work because it doesn’t maintain the proper order of the elements in the scattered vectors.
>
> I attach a modified version of the test code where I put elements into the global vector, then carry out the scatter, and check on the subcomms that they are correct.
>
> You can see everything is fine with the AOApplicationToPetsc calls, but the comparison fails when those are commented out.
>
> If there is some way I can achieve the right VecScatters without those calls, I would be happy to know how to do that.
>
> Thank you again for your help.
>
> Randy
>
> ps. I suggest you run this test with nx=ny=nz=10 and only a couple subcomms and maybe 4 processes to demonstrate the behavior
>
>
>> On Apr 20, 2020, at 2:45 PM, Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>
>> Hello, Randy,
>> I further looked at the problem and believe it was due to overwhelming traffic. The code sometimes fails at MPI_Waitall. I printed out MPI error strings of bad MPI Statuses. One of them is like "MPID_nem_tcp_connpoll(1845): Communication error with rank 25: Connection reset by peer", which is a tcp error and has nothing to do with petsc.
>> Further investigation shows in the case of 5120 ranks with 320 sub communicators, during VecScatterSetUp, each rank has around 640 isends/irecvs neighbors, and quite a few ranks has 1280 isends neighbors. I guess these overwhelming isends occasionally crashed the connection.
>> The piece of code in VecScatterSetUp is to calculate the communication pattern. With index sets "having good locality", the calculate itself incurs less traffic. Here good locality means indices in an index set mostly point to local entries. However, the AOApplicationToPetsc() call in your code unnecessarily ruined the good petsc ordering. If we remove AOApplicationToPetsc() (the vecscatter result is still correct) , then each rank uniformly has around 320 isends/irecvs.
>> So, test with this modification and see if it really works in your environment. If not applicable, we can provide options in petsc to carry out the communication in phases to avoid flooding the network (though it is better done by MPI).
>>
>> Thanks.
>> --Junchao Zhang
>>
>>
>> On Fri, Apr 17, 2020 at 10:47 AM Randall Mackie <rlmackie862 at gmail.com <mailto:rlmackie862 at gmail.com>> wrote:
>> Hi Junchao,
>>
>> Thank you for your efforts.
>> We tried petsc-3.13.0 but it made no difference.
>> We think now the issue are with sysctl parameters, and increasing those seemed to have cleared up the problem.
>> This also most likely explains how different clusters had different behaviors with our test code.
>>
>> We are now running our code and will report back once we are sure that there are no further issues.
>>
>> Thanks again for your help.
>>
>> Randy M.
>>
>>> On Apr 17, 2020, at 8:09 AM, Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>>
>>>
>>>
>>>
>>> On Thu, Apr 16, 2020 at 11:13 PM Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>> Randy,
>>> I reproduced your error with petsc-3.12.4 and 5120 mpi ranks. I also found the error went away with petsc-3.13. However, I have not figured out what is the bug and which commit fixed it :).
>>> So at your side, it is better to use the latest petsc.
>>> I want to add that even with petsc-3.12.4 the error is random. I was only able to reproduce the error once, so I can not claim petsc-3.13 actually fixed it (or, the bug is really in petsc).
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Thu, Apr 16, 2020 at 9:06 PM Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>> Randy,
>>> Up to now I could not reproduce your error, even with the biggest mpirun -n 5120 ./test -nsubs 320 -nx 100 -ny 100 -nz 100
>>> While I continue doing test, you can try other options. It looks you want to duplicate a vector to subcomms. I don't think you need the two lines:
>>> call AOApplicationToPetsc(aoParent,nis,ind1,ierr)
>>> call AOApplicationToPetsc(aoSub,nis,ind2,ierr)
>>> In addition, you can use simpler and more memory-efficient index sets. There is a petsc example for this task, see case 3 in https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c <https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vscat/tests/ex9.c>
>>> BTW, it is good to use petsc master so we are on the same page.
>>> --Junchao Zhang
>>>
>>>
>>> On Wed, Apr 15, 2020 at 10:28 AM Randall Mackie <rlmackie862 at gmail.com <mailto:rlmackie862 at gmail.com>> wrote:
>>> Hi Junchao,
>>>
>>> So I was able to create a small test code that duplicates the issue we have been having, and it is attached to this email in a zip file.
>>> Included is the test.F90 code, the commands to duplicate crash and to duplicate a successful run, output errors, and our petsc configuration.
>>>
>>> Our findings to date include:
>>>
>>> The error is reproducible in a very short time with this script
>>> It is related to nproc*nsubs and (although to a less extent) to DM grid size
>>> It happens regardless of MPI implementation (mpich, intel mpi 2018, 2019, openmpi) or compiler (gfortran/gcc , intel 2018)
>>> No effect changing vecscatter_type to mpi1 or mpi3. Mpi1 seems to slightly increase the limit, but still fails on the full machine set.
>>> Nothing looks interesting on valgrind
>>>
>>> Our initial tests were carried out on an Azure cluster, but we also tested on our smaller cluster, and we found the following:
>>>
>>> Works:
>>> $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 1280 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100
>>>
>>> Crashes (this works on Azure)
>>> $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 2560 -hostfile hostfile ./test -nsubs 80 -nx 100 -ny 100 -nz 100
>>>
>>> So it looks like it may also be related to the physical number of nodes as well.
>>>
>>> In any case, even with 2560 processes on 192 cores the memory does not go above 3.5 Gbyes so you don’t need a huge cluster to test.
>>>
>>> Thanks,
>>>
>>> Randy M.
>>>
>>>
>>>
>>>> On Apr 14, 2020, at 12:23 PM, Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>>>
>>>> There is an MPI_Allreduce in PetscGatherNumberOfMessages, that is why I doubted it was the problem. Even if users configure petsc with 64-bit indices, we use PetscMPIInt in MPI calls. So it is not a problem.
>>>> Try -vecscatter_type mpi1 to restore to the original VecScatter implementation. If the problem still remains, could you provide a test example for me to debug?
>>>>
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Tue, Apr 14, 2020 at 12:13 PM Randall Mackie <rlmackie862 at gmail.com <mailto:rlmackie862 at gmail.com>> wrote:
>>>> Hi Junchao,
>>>>
>>>> We have tried your two suggestions but the problem remains.
>>>> And the problem seems to be on the MPI_Isend line 117 in PetscGatherMessageLengths and not MPI_AllReduce.
>>>>
>>>> We have now tried Intel MPI, Mpich, and OpenMPI, and so are thinking the problem must be elsewhere and not MPI.
>>>>
>>>> Give that this is a 64 bit indices build of PETSc, is there some possible incompatibility between PETSc and MPI calls?
>>>>
>>>> We are open to any other possible suggestions to try as other than valgrind on thousands of processes we seem to have run out of ideas.
>>>>
>>>> Thanks, Randy M.
>>>>
>>>>> On Apr 13, 2020, at 8:54 AM, Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>>>>
>>>>>
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Mon, Apr 13, 2020 at 10:53 AM Junchao Zhang <junchao.zhang at gmail.com <mailto:junchao.zhang at gmail.com>> wrote:
>>>>> Randy,
>>>>> Someone reported similar problem before. It turned out an Intel MPI MPI_Allreduce bug. A workaround is setting the environment variable I_MPI_ADJUST_ALLREDUCE=1.arr
>>>>> Correct: I_MPI_ADJUST_ALLREDUCE=1
>>>>> But you mentioned mpich also had the error. So maybe the problem is not the same. So let's try the workaround first. If it doesn't work, add another petsc option -build_twosided allreduce, which is a workaround for Intel MPI_Ibarrier bugs we met.
>>>>> Thanks.
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Mon, Apr 13, 2020 at 10:38 AM Randall Mackie <rlmackie862 at gmail.com <mailto:rlmackie862 at gmail.com>> wrote:
>>>>> Dear PETSc users,
>>>>>
>>>>> We are trying to understand an issue that has come up in running our code on a large cloud cluster with a large number of processes and subcomms.
>>>>> This is code that we use daily on multiple clusters without problems, and that runs valgrind clean for small test problems.
>>>>>
>>>>> The run generates the following messages, but doesn’t crash, just seems to hang with all processes continuing to show activity:
>>>>>
>>>>> [492]PETSC ERROR: #1 PetscGatherMessageLengths() line 117 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/sys/utils/mpimesg.c
>>>>> [492]PETSC ERROR: #2 VecScatterSetUp_SF() line 658 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/impls/sf/vscatsf.c
>>>>> [492]PETSC ERROR: #3 VecScatterSetUp() line 209 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscatfce.c
>>>>> [492]PETSC ERROR: #4 VecScatterCreate() line 282 in /mnt/home/cgg/PETSc/petsc-3.12.4/src/vec/vscat/interface/vscreate.c
>>>>>
>>>>>
>>>>> Looking at line 117 in PetscGatherMessageLengths we find the offending statement is the MPI_Isend:
>>>>>
>>>>>
>>>>> /* Post the Isends with the message length-info */
>>>>> for (i=0,j=0; i<size; ++i) {
>>>>> if (ilengths[i]) {
>>>>> ierr = MPI_Isend((void*)(ilengths+i),1,MPI_INT,i,tag,comm,s_waits+j);CHKERRQ(ierr);
>>>>> j++;
>>>>> }
>>>>> }
>>>>>
>>>>> We have tried this with Intel MPI 2018, 2019, and mpich, all giving the same problem.
>>>>>
>>>>> We suspect there is some limit being set on this cloud cluster on the number of file connections or something, but we don’t know.
>>>>>
>>>>> Anyone have any ideas? We are sort of grasping for straws at this point.
>>>>>
>>>>> Thanks, Randy M.
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200430/e904b47e/attachment-0001.html>
More information about the petsc-users
mailing list