[petsc-users] Can anyone guess the possible reason of the following errors?

[Barry Smith]

  Perhaps some bad data is being passed into VecScatterCreate(). I would suggest having SpmcsSFCreateVecScatter
 validate the IS's and Vecs being passed in. For example, do the IS have tons of duplicates, how long are they etc…

   Barry

On Dec 3, 2012, at 1:12 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:

> More details for the errors:
> 
> [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in src/sys/utils/mpimesg.c
> [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in src/vec/vec/utils/vpscat.c
> [0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: SpmcsSFCreateVecScatter() line 96 in SpmcsSFComm.cpp
> [0]PETSC ERROR: moveDataBetweenRootsAndLeaves() line 133 in SpmcsSFComm.cpp
> [0]PETSC ERROR: SpmcsSFCreateNormalizedEmbeddedSF() line 359 in SpmcsSFComm.cpp
> [0]PETSC ERROR: SpmcsSFDistributeSection() line 343 in SpmcsSection.cpp
> [0]PETSC ERROR: SpmcsMeshDistribute() line 444 in distributeMesh.cpp
> [0]PETSC ERROR: DMmeshInitialize() line 32 in mgInitialize.cpp
> [0]PETSC ERROR: main() line 64 in linearElasticity3d.cpp
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> 
> =====================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   EXIT CODE: 256
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> =====================================================================================
> [proxy:0:1 at node1778] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:1 at node1778] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:1 at node1778] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:2 at node1777] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:2 at node1777] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:2 at node1777] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:3 at node1773] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:3 at node1773] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:3 at node1773] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:4 at node1770] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:4 at node1770] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:4 at node1770] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:6 at node1760] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:6 at node1760] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:6 at node1760] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:7 at node1758] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:7 at node1758] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:7 at node1758] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:8 at node1738] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:8 at node1738] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:8 at node1738] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:9 at node1736] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:9 at node1736] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:9 at node1736] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:10 at node1668] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:10 at node1668] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:10 at node1668] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:11 at node1667] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:11 at node1667] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:11 at node1667] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:12 at node1658] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:12 at node1658] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:12 at node1658] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:13 at node1656] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:13 at node1656] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:13 at node1656] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:14 at node1637] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:14 at node1637] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:14 at node1637] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:15 at node1636] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:15 at node1636] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:15 at node1636] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:16 at node1611] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:16 at node1611] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:16 at node1611] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:17 at node1380] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:17 at node1380] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:17 at node1380] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:18 at node1379] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:18 at node1379] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:18 at node1379] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:19 at node1378] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:19 at node1378] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:19 at node1378] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:20 at node1377] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:20 at node1377] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:20 at node1377] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:21 at node1376] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:21 at node1376] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:21 at node1376] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:22 at node1375] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:22 at node1375] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:22 at node1375] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:23 at node1374] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:23 at node1374] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:23 at node1374] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:24 at node1373] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:24 at node1373] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:24 at node1373] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:25 at node1372] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:25 at node1372] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:25 at node1372] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:26 at node1371] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:26 at node1371] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:26 at node1371] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:27 at node1370] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:27 at node1370] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:27 at node1370] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:28 at node1369] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:28 at node1369] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:28 at node1369] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:29 at node1368] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:29 at node1368] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:29 at node1368] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:30 at node1367] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:30 at node1367] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:30 at node1367] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [proxy:0:31 at node1366] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> [proxy:0:31 at node1366] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:31 at node1366] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> [mpiexec at node1780] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated badly; aborting
> [mpiexec at node1780] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
> [mpiexec at node1780] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:199): launcher returned error waiting for completion
> [mpiexec at node1780] main (./ui/mpich/mpiexec.c:385): process manager error waiting for completion
> 
> It seems nothing. 
> 
> On Mon, Dec 3, 2012 at 11:41 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Mon, Dec 3, 2012 at 12:38 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> > Hi all,
> >
> > Can anyone guess the possible reason of the following errors:
> >
> >
> > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
> > src/sys/utils/mpimesg.c
> > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
> > src/vec/vec/utils/vpscat.c
> > [0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
> 
> Partial error messages are generally not helpful.
> 
>    Matt
> 
> > I have been working for several days to figure out the reason, but now I
> > still get nothing.  I use Petsc-3.3-p3 based on the mvapich2-1.6. I tried to
> > use vecscatter to distribute the mesh. When the mesh was small, everything
> > was ok. But when the mesh became larger about 14,000,000 elements, I got the
> > above errors.
> >
> > --
> > Fande Kong
> > Department of Computer Science
> > University of Colorado at Boulder
> >
> >
> 
> 
> 
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> Fande Kong
> ShenZhen Institutes of Advanced Technology
> Chinese Academy of Sciences
>