[petsc-users] Petsc cannot be initialized on vesta in some --mode options

[Jeff Hammond]

On Wed, Jan 22, 2014 at 12:24 PM, Jed Brown <jed at jedbrown.org> wrote:
> Roc Wang <pengxwang at hotmail.com> writes:
>> The configure.log is attached.
>> I also tried running the program with command like:
>> qsub -n <number of nodes> -t 10 --mode <ranks per node> --env "F00=a:BAR=b:PAMID_COLLECTIVES=0" ./x.r
>>
>> For 1024 as the total number of ranks, the program was able to run in
>> c1, c16, c32, and c64 mode, but still petsc cannot be initialized in
>> c2, c4, and c8 mode. The runtime log files for c8 mode were
>> attached. Thanks.
>
> Hmm, appears to be this line erroring.
>
>       ierr     = MPI_Bcast(&len,1,MPIU_SIZE_T,0,PETSC_COMM_WORLD);CHKERRQ(ierr);
>
> But at least some processes are stuck in an earlier
> PetscOptionsInsertFile (which contains MPI_Bcast).  This is like the
> problem we have seen before, but it used to go away with
> PAMID_COLLECTIVES=0.  Can you run with -skip_petscrc?  That will skip a
> couple MPI_Bcasts and might get it working again.

I've always thought that MPI_Bcast wasn't actually the problem anyways
but rather it was the POSIX calls that were breaking PETSc_Init.

> Jeff, has the Vesta stack changed recently?  Any reason why
> PAMID_COLLECTIVES=0 would not be taking effect here?

No and no.

Jeff

-- 
Jeff Hammond
jeff.science at gmail.com