[petsc-users] Can anyone guess the possible reason of the following errors?

[Fande Kong]

More details for the errors:

[0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
src/sys/utils/mpimesg.c
[0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
src/vec/vec/utils/vpscat.c
[0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
[0]PETSC ERROR: SpmcsSFCreateVecScatter() line 96 in SpmcsSFComm.cpp
[0]PETSC ERROR: moveDataBetweenRootsAndLeaves() line 133 in SpmcsSFComm.cpp
[0]PETSC ERROR: SpmcsSFCreateNormalizedEmbeddedSF() line 359 in
SpmcsSFComm.cpp
[0]PETSC ERROR: SpmcsSFDistributeSection() line 343 in SpmcsSection.cpp
[0]PETSC ERROR: SpmcsMeshDistribute() line 444 in distributeMesh.cpp
[0]PETSC ERROR: DMmeshInitialize() line 32 in mgInitialize.cpp
[0]PETSC ERROR: main() line 64 in linearElasticity3d.cpp
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

=====================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 256
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
[proxy:0:1 at node1778] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:1 at node1778] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:1 at node1778] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:2 at node1777] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:2 at node1777] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:2 at node1777] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:3 at node1773] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:3 at node1773] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:3 at node1773] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:4 at node1770] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:4 at node1770] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:4 at node1770] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:6 at node1760] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:6 at node1760] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:6 at node1760] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:7 at node1758] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:7 at node1758] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:7 at node1758] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:8 at node1738] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:8 at node1738] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:8 at node1738] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:9 at node1736] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:9 at node1736] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:9 at node1736] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:10 at node1668] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:10 at node1668] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:10 at node1668] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:11 at node1667] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:11 at node1667] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:11 at node1667] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:12 at node1658] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:12 at node1658] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:12 at node1658] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:13 at node1656] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:13 at node1656] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:13 at node1656] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:14 at node1637] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:14 at node1637] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:14 at node1637] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:15 at node1636] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:15 at node1636] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:15 at node1636] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:16 at node1611] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:16 at node1611] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:16 at node1611] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:17 at node1380] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:17 at node1380] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:17 at node1380] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:18 at node1379] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:18 at node1379] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:18 at node1379] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:19 at node1378] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:19 at node1378] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:19 at node1378] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:20 at node1377] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:20 at node1377] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:20 at node1377] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:21 at node1376] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:21 at node1376] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:21 at node1376] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:22 at node1375] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:22 at node1375] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:22 at node1375] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:23 at node1374] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:23 at node1374] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:23 at node1374] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:24 at node1373] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:24 at node1373] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:24 at node1373] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:25 at node1372] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:25 at node1372] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:25 at node1372] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:26 at node1371] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:26 at node1371] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:26 at node1371] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:27 at node1370] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:27 at node1370] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:27 at node1370] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:28 at node1369] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:28 at node1369] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:28 at node1369] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:29 at node1368] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:29 at node1368] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:29 at node1368] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:30 at node1367] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:30 at node1367] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:30 at node1367] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[proxy:0:31 at node1366] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
[proxy:0:31 at node1366] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:31 at node1366] main (./pm/pmiserv/pmip.c:214): demux engine error
waiting for event
[mpiexec at node1780] HYDT_bscu_wait_for_completion
(./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
badly; aborting
[mpiexec at node1780] HYDT_bsci_wait_for_completion
(./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
completion
[mpiexec at node1780] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:199): launcher returned error waiting for
completion
[mpiexec at node1780] main (./ui/mpich/mpiexec.c:385): process manager error
waiting for completion

It seems nothing.

On Mon, Dec 3, 2012 at 11:41 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Dec 3, 2012 at 12:38 PM, Fande Kong <fande.kong at colorado.edu>
> wrote:
> > Hi all,
> >
> > Can anyone guess the possible reason of the following errors:
> >
> >
> > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
> > src/sys/utils/mpimesg.c
> > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
> > src/vec/vec/utils/vpscat.c
> > [0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
>
> Partial error messages are generally not helpful.
>
>    Matt
>
> > I have been working for several days to figure out the reason, but now I
> > still get nothing.  I use Petsc-3.3-p3 based on the mvapich2-1.6. I
> tried to
> > use vecscatter to distribute the mesh. When the mesh was small,
> everything
> > was ok. But when the mesh became larger about 14,000,000 elements, I got
> the
> > above errors.
> >
> > --
> > Fande Kong
> > Department of Computer Science
> > University of Colorado at Boulder
> >
> >
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
>


-- 
Fande Kong
ShenZhen Institutes of Advanced Technology
Chinese Academy of Sciences
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121203/dce319b8/attachment-0001.html>