[petsc-users] Can anyone guess the possible reason of the following errors?

Mon Dec 3 13:25:01 CST 2012

On Mon, Dec 3, 2012 at 1:23 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> Are there any constraints for IS and Vec?

No, but this appears to be inconsistency.

  Matt

> On Mon, Dec 3, 2012 at 12:19 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>   Perhaps some bad data is being passed into VecScatterCreate(). I would
>> suggest having SpmcsSFCreateVecScatter
>>  validate the IS's and Vecs being passed in. For example, do the IS have
>> tons of duplicates, how long are they etc…
>>
>>    Barry
>>
>> On Dec 3, 2012, at 1:12 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>>
>> > More details for the errors:
>> >
>> > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
>> > src/sys/utils/mpimesg.c
>> > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
>> > src/vec/vec/utils/vpscat.c
>> > [0]PETSC ERROR: VecScatterCreate() line 1431 in
>> > src/vec/vec/utils/vscat.c
>> > [0]PETSC ERROR: SpmcsSFCreateVecScatter() line 96 in SpmcsSFComm.cpp
>> > [0]PETSC ERROR: moveDataBetweenRootsAndLeaves() line 133 in
>> > SpmcsSFComm.cpp
>> > [0]PETSC ERROR: SpmcsSFCreateNormalizedEmbeddedSF() line 359 in
>> > SpmcsSFComm.cpp
>> > [0]PETSC ERROR: SpmcsSFDistributeSection() line 343 in SpmcsSection.cpp
>> > [0]PETSC ERROR: SpmcsMeshDistribute() line 444 in distributeMesh.cpp
>> > [0]PETSC ERROR: DMmeshInitialize() line 32 in mgInitialize.cpp
>> > [0]PETSC ERROR: main() line 64 in linearElasticity3d.cpp
>> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>> >
>> >
>> > =====================================================================================
>> > =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> > =   EXIT CODE: 256
>> > =   CLEANING UP REMAINING PROCESSES
>> > =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>> >
>> > =====================================================================================
>> > [proxy:0:1 at node1778] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:1 at node1778] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:1 at node1778] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:2 at node1777] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:2 at node1777] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:2 at node1777] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:3 at node1773] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:3 at node1773] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:3 at node1773] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:4 at node1770] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:4 at node1770] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:4 at node1770] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:6 at node1760] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:6 at node1760] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:6 at node1760] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:7 at node1758] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:7 at node1758] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:7 at node1758] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:8 at node1738] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:8 at node1738] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:8 at node1738] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:9 at node1736] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:9 at node1736] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:9 at node1736] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:10 at node1668] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:10 at node1668] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:10 at node1668] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:11 at node1667] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:11 at node1667] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:11 at node1667] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:12 at node1658] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:12 at node1658] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:12 at node1658] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:13 at node1656] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:13 at node1656] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:13 at node1656] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:14 at node1637] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:14 at node1637] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:14 at node1637] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:15 at node1636] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:15 at node1636] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:15 at node1636] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:16 at node1611] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:16 at node1611] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:16 at node1611] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:17 at node1380] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:17 at node1380] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:17 at node1380] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:18 at node1379] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:18 at node1379] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:18 at node1379] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:19 at node1378] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:19 at node1378] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:19 at node1378] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:20 at node1377] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:20 at node1377] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:20 at node1377] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:21 at node1376] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:21 at node1376] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:21 at node1376] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:22 at node1375] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:22 at node1375] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:22 at node1375] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:23 at node1374] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:23 at node1374] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:23 at node1374] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:24 at node1373] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:24 at node1373] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:24 at node1373] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:25 at node1372] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:25 at node1372] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:25 at node1372] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:26 at node1371] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:26 at node1371] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:26 at node1371] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:27 at node1370] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:27 at node1370] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:27 at node1370] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:28 at node1369] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:28 at node1369] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:28 at node1369] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:29 at node1368] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:29 at node1368] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:29 at node1368] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:30 at node1367] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:30 at node1367] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:30 at node1367] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [proxy:0:31 at node1366] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> > [proxy:0:31 at node1366] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:31 at node1366] main (./pm/pmiserv/pmip.c:214): demux engine error
>> > waiting for event
>> > [mpiexec at node1780] HYDT_bscu_wait_for_completion
>> > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
>> > badly; aborting
>> > [mpiexec at node1780] HYDT_bsci_wait_for_completion
>> > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
>> > completion
>> > [mpiexec at node1780] HYD_pmci_wait_for_completion
>> > (./pm/pmiserv/pmiserv_pmci.c:199): launcher returned error waiting for
>> > completion
>> > [mpiexec at node1780] main (./ui/mpich/mpiexec.c:385): process manager
>> > error waiting for completion
>> >
>> > It seems nothing.
>> >
>> > On Mon, Dec 3, 2012 at 11:41 AM, Matthew Knepley <knepley at gmail.com>
>> > wrote:
>> > On Mon, Dec 3, 2012 at 12:38 PM, Fande Kong <fande.kong at colorado.edu>
>> > wrote:
>> > > Hi all,
>> > >
>> > > Can anyone guess the possible reason of the following errors:
>> > >
>> > >
>> > > [0]PETSC ERROR: PetscGatherMessageLengths() line 133 in
>> > > src/sys/utils/mpimesg.c
>> > > [0]PETSC ERROR: VecScatterCreate_PtoP() line 2188 in
>> > > src/vec/vec/utils/vpscat.c
>> > > [0]PETSC ERROR: VecScatterCreate() line 1431 in
>> > > src/vec/vec/utils/vscat.c
>> >
>> > Partial error messages are generally not helpful.
>> >
>> >    Matt
>> >
>> > > I have been working for several days to figure out the reason, but now
>> > > I
>> > > still get nothing.  I use Petsc-3.3-p3 based on the mvapich2-1.6. I
>> > > tried to
>> > > use vecscatter to distribute the mesh. When the mesh was small,
>> > > everything
>> > > was ok. But when the mesh became larger about 14,000,000 elements, I
>> > > got the
>> > > above errors.
>> > >
>> > > --
>> > > Fande Kong
>> > > Department of Computer Science
>> > > University of Colorado at Boulder
>> > >
>> > >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments is infinitely more interesting than any results to which
>> > their experiments lead.
>> > -- Norbert Wiener
>> >
>> >
>> >
>> >
>> > --
>> > Fande Kong
>> > ShenZhen Institutes of Advanced Technology
>> > Chinese Academy of Sciences
>> >
>>
>>
>
>
>
> --
> Fande Kong
> ShenZhen Institutes of Advanced Technology
> Chinese Academy of Sciences
>

--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener