<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'><br><br><div>> From: jed@jedbrown.org<br>> To: pengxwang@hotmail.com; petsc-users@mcs.anl.gov<br>> CC: jeff.science@gmail.com<br>> Subject: RE: [petsc-users] Petsc cannot be initialized on vesta in some --mode options<br>> Date: Wed, 22 Jan 2014 11:24:21 -0700<br>> <br>> Roc Wang <pengxwang@hotmail.com> writes:<br>> > The configure.log is attached.<br>> > I also tried running the program with command like:<br>> > qsub -n <number of nodes> -t 10 --mode <ranks per node> --env "F00=a:BAR=b:PAMID_COLLECTIVES=0" ./x.r <br>> ><br>> > For 1024 as the total number of ranks, the program was able to run in<br>> > c1, c16, c32, and c64 mode, but still petsc cannot be initialized in<br>> > c2, c4, and c8 mode. The runtime log files for c8 mode were<br>> > attached. Thanks.<br>> <br>> Hmm, appears to be this line erroring.<br>> <br>> ierr = MPI_Bcast(&len,1,MPIU_SIZE_T,0,PETSC_COMM_WORLD);CHKERRQ(ierr);<br>> <br>> But at least some processes are stuck in an earlier<br>> PetscOptionsInsertFile (which contains MPI_Bcast). This is like the<br>> problem we have seen before, but it used to go away with<br>> PAMID_COLLECTIVES=0. Can you run with -skip_petscrc? That will skip a<br>> couple MPI_Bcasts and might get it working again.<br>> <br>> <br>> Jeff, has the Vesta stack changed recently? Any reason why<br>> PAMID_COLLECTIVES=0 would not be taking effect here?<br>> <br>> I'll try to reproduce Roc's findings later today.<br><br>Yeah. The program works with -skip_petscrc. I am trying to run the full PETSc solver and take a look what will happen.<br><br>I just recalled there was an error caused by calling MPI_Bcast() when I transferred the program from other machine to vesta. At that time I just replace the MPI_Bcast(). I am not sure whether it is the same problem as what we have here. But just for your information.<br></div> </div></body>
</html>