<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.30.3">
</HEAD>
<BODY>
Darius,<BR>
<BR>
Thanks for the help. It's works. I forgot to recompile my application.<BR>
<BR>
But I have another question. It's possible use the checkpoint-restart feature in mpich2 using slurm pm?<BR>
<BR>
I tried execute <BR>
<BR>
salloc -n 26 mpiexec -ckpointlib blcr -ckpoint-prefix ./teste.ckpoint -ckpoint-interval 30 Dyna Prea_teste001.p3d 2<BR>
<BR>
And to restart I use<BR>
<BR>
salloc -n 26 mpiexec -ckpointlib blcr -ckpoint-prefix ./teste.ckpoint -ckpoint-interval 30 -ckpoint-num 2<BR>
<BR>
I received the follow message<BR>
salloc: Granted job allocation 12613<BR>
[<A HREF="mailto:mpiexec@masternode1">mpiexec@masternode1</A>] HYD_pmcd_pmi_alloc_pg_scratch (./pm/pmiserv/pmiserv_utils.c:594): assert (pg->pg_process_count * sizeof(struct HYD_pmcd_pmi_ecount)) failed<BR>
[<A HREF="mailto:mpiexec@masternode1">mpiexec@masternode1</A>] HYD_pmci_launch_procs (./pm/pmiserv/pmiserv_pmci.c:103): error allocating pg scratch space<BR>
[<A HREF="mailto:mpiexec@masternode1">mpiexec@masternode1</A>] main (./ui/mpich/mpiexec.c:401): process manager returned error launching processes<BR>
salloc: Relinquishing job allocation 12613<BR>
<BR>
The entire cluster are run under NFS.<BR>
<BR>
But if I use salloc to select the nodes and I use the -f hosts (with the nodes allocated by salloc) works perfectly. <BR>
<BR>
Regards<BR>
<BR>
Fernando Luz<BR>
<BR>
<BR>
On Seg, 2011-10-24 at 16:41 -0500, Darius Buntinas wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
Hmm strange. Did you do a make clean first? I.e.:
make clean
make
make install
Also make sure you recompile your app (maybe even do a make clean for the app too).
-d
On Oct 22, 2011, at 3:28 PM, Fernando Luz wrote:
> Hi Darius,
>
> I applied the patch, but I have the same errors.
>
> Do you need some file or info about my system?
>
> Regards
>
> Fernando Luz
>
> ----- Mensagem original -----
> De: "Darius Buntinas" <<A HREF="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</A>>
> Para: <A HREF="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A>
> Enviadas: Sexta-feira, 21 de Outubro de 2011 16:20:49
> Assunto: Re: [mpich-discuss] Trouble with checkpoint
>
> Hi Fernando,
>
> Can you apply this patch and see if it fixes your problem?
>
> Let us know how it goes.
> -d
>
>
>
> On Oct 19, 2011, at 2:13 PM, Fernando Luz wrote:
>
>> Hi,
>>
>> I tried use the checkpoint-restart with this execution.
>>
>> mpiexec -ckpointlib blcr -ckpoint-prefix ./teste.ckpoint -ckpoint-interval 30 -f hosts -n 26 Dyna Prea_teste001.p3d 2
>>
>> with mpich2 and I received the follows errors.
>>
>> 0% [= ] 00:00:28 / 00:56:27
>> [proxy:0:<A HREF="mailto:0@s23n20.gradebr.tpn">0@s23n20.gradebr.tpn</A>] requesting checkpoint
>> [proxy:0:<A HREF="mailto:1@s23n21.gradebr.tpn">1@s23n21.gradebr.tpn</A>] requesting checkpoint
>> [proxy:0:<A HREF="mailto:2@s23n22.gradebr.tpn">2@s23n22.gradebr.tpn</A>] requesting checkpoint
>> [proxy:0:<A HREF="mailto:3@s23n23.gradebr.tpn">3@s23n23.gradebr.tpn</A>] requesting checkpoint
>> [proxy:0:<A HREF="mailto:0@s23n20.gradebr.tpn">0@s23n20.gradebr.tpn</A>] checkpoint completed
>> [proxy:0:<A HREF="mailto:1@s23n21.gradebr.tpn">1@s23n21.gradebr.tpn</A>] checkpoint completed
>> [proxy:0:<A HREF="mailto:2@s23n22.gradebr.tpn">2@s23n22.gradebr.tpn</A>] checkpoint completed
>> [proxy:0:<A HREF="mailto:3@s23n23.gradebr.tpn">3@s23n23.gradebr.tpn</A>] checkpoint completed
>> 0% [= ] 00:00:29 / 00:56:28Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1ebebfc0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff7ba7b620) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1f84f600, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff957566a0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1fc58d50, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7ffff54102a0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x7752ca0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fffeab72ca0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x12274ca0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fffb55e4ea0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1b6c4600, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff74e63520) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x15511ca0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff9fb57ca0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x815afc0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff87e31fa0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0xf1e7d80, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff19d30120) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1758f9a0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff06ac13a0) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0xaaf8ce0, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff7e488920) failed
>> MPIDI_CH3I_Progress(321)..:
>> MPIDI_nem_ckpt_finish(469): sem_wait() failed Interrupted system call
>> Fatal error in MPI_Recv: Other MPI error, error stack:
>> MPI_Recv(186).............: MPI_Recv(buf=0x1cc47990, count=7, MPI_DOUBLE, src=0, tag=2, MPI_COMM_WORLD, status=0x7fff00638520) failed
>>
>> without checkpoint, the execution is accomplish. How I need to proceed to solve this error?
>>
>> Regards
>>
>> Fernando Luz
>> _______________________________________________
>> mpich-discuss mailing list <A HREF="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A>
>> To manage subscription options or unsubscribe:
>> <A HREF="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</A>
>
>
> _______________________________________________
> mpich-discuss mailing list <A HREF="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A>
> To manage subscription options or unsubscribe:
> <A HREF="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</A>
> _______________________________________________
> mpich-discuss mailing list <A HREF="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A>
> To manage subscription options or unsubscribe:
> <A HREF="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</A>
_______________________________________________
mpich-discuss mailing list <A HREF="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A>
To manage subscription options or unsubscribe:
<A HREF="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</A>
</PRE>
</BLOCKQUOTE>
<BR>
</BODY>
</HTML>