[petsc-users] Can anyone guess the possible reason of the following errors?

Mon Dec 3 13:27:54 CST 2012

On Dec 3, 2012, at 1:23 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:

> Are there any constraints for IS and Vec?

   You could also run with the option -mpi_return_on_error false 

and MPI may print an error message of what it thinks has gone wrong.

   Barry

> 
> On Mon, Dec 3, 2012 at 12:19 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Perhaps some bad data is being passed into VecScatterCreate(). I would suggest having SpmcsSFCreateVecScatter
>  validate the IS's and Vecs being passed in. For example, do the IS have tons of duplicates, how long are they etc…
> 
>    Barry
> 
> On Dec 3, 2012, at 1:12 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> 
> > More details for the errors:
> >
> > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in src/sys/utils/mpimesg.c
> > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in src/vec/vec/utils/vpscat.c
> > [0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
> > [0]PETSC ERROR: SpmcsSFCreateVecScatter() line 96 in SpmcsSFComm.cpp
> > [0]PETSC ERROR: moveDataBetweenRootsAndLeaves() line 133 in SpmcsSFComm.cpp
> > [0]PETSC ERROR: SpmcsSFCreateNormalizedEmbeddedSF() line 359 in SpmcsSFComm.cpp
> > [0]PETSC ERROR: SpmcsSFDistributeSection() line 343 in SpmcsSection.cpp
> > [0]PETSC ERROR: SpmcsMeshDistribute() line 444 in distributeMesh.cpp
> > [0]PETSC ERROR: DMmeshInitialize() line 32 in mgInitialize.cpp
> > [0]PETSC ERROR: main() line 64 in linearElasticity3d.cpp
> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> >
> > =====================================================================================
> > =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> > =   EXIT CODE: 256
> > =   CLEANING UP REMAINING PROCESSES
> > =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> > =====================================================================================
> > [proxy:0:1 at node1778] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:1 at node1778] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:1 at node1778] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:2 at node1777] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:2 at node1777] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:2 at node1777] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:3 at node1773] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:3 at node1773] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:3 at node1773] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:4 at node1770] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:4 at node1770] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:4 at node1770] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:6 at node1760] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:6 at node1760] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:6 at node1760] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:7 at node1758] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:7 at node1758] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:7 at node1758] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:8 at node1738] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:8 at node1738] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:8 at node1738] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:9 at node1736] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:9 at node1736] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:9 at node1736] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:10 at node1668] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:10 at node1668] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:10 at node1668] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:11 at node1667] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:11 at node1667] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:11 at node1667] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:12 at node1658] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:12 at node1658] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:12 at node1658] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:13 at node1656] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:13 at node1656] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:13 at node1656] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:14 at node1637] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:14 at node1637] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:14 at node1637] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:15 at node1636] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:15 at node1636] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:15 at node1636] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:16 at node1611] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:16 at node1611] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:16 at node1611] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:17 at node1380] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:17 at node1380] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:17 at node1380] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:18 at node1379] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:18 at node1379] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:18 at node1379] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:19 at node1378] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:19 at node1378] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:19 at node1378] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:20 at node1377] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:20 at node1377] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:20 at node1377] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:21 at node1376] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:21 at node1376] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:21 at node1376] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:22 at node1375] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:22 at node1375] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:22 at node1375] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:23 at node1374] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:23 at node1374] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:23 at node1374] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:24 at node1373] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:24 at node1373] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:24 at node1373] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:25 at node1372] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:25 at node1372] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:25 at node1372] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:26 at node1371] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:26 at node1371] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:26 at node1371] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:27 at node1370] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:27 at node1370] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:27 at node1370] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:28 at node1369] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:28 at node1369] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:28 at node1369] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:29 at node1368] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:29 at node1368] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:29 at node1368] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:30 at node1367] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:30 at node1367] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:30 at node1367] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [proxy:0:31 at node1366] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> > [proxy:0:31 at node1366] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
> > [proxy:0:31 at node1366] main (./pm/pmiserv/pmip.c:214): demux engine error waiting for event
> > [mpiexec at node1780] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated badly; aborting
> > [mpiexec at node1780] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
> > [mpiexec at node1780] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:199): launcher returned error waiting for completion
> > [mpiexec at node1780] main (./ui/mpich/mpiexec.c:385): process manager error waiting for completion
> >
> > It seems nothing.
> >
> > On Mon, Dec 3, 2012 at 11:41 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Mon, Dec 3, 2012 at 12:38 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> > > Hi all,
> > >
> > > Can anyone guess the possible reason of the following errors:
> > >
> > >
> > > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
> > > src/sys/utils/mpimesg.c
> > > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
> > > src/vec/vec/utils/vpscat.c
> > > [0]PETSC ERROR: VecScatterCreate() line 1431 in src/vec/vec/utils/vscat.c
> >
> > Partial error messages are generally not helpful.
> >
> >    Matt
> >
> > > I have been working for several days to figure out the reason, but now I
> > > still get nothing.  I use Petsc-3.3-p3 based on the mvapich2-1.6. I tried to
> > > use vecscatter to distribute the mesh. When the mesh was small, everything
> > > was ok. But when the mesh became larger about 14,000,000 elements, I got the
> > > above errors.
> > >
> > > --
> > > Fande Kong
> > > Department of Computer Science
> > > University of Colorado at Boulder
> > >
> > >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > Fande Kong
> > ShenZhen Institutes of Advanced Technology
> > Chinese Academy of Sciences
> >
> 
> 
> 
> 
> 
> -- 
> Fande Kong
> ShenZhen Institutes of Advanced Technology
> Chinese Academy of Sciences
> 