<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'><br><br><div>> From: jed@jedbrown.org<br>> To: pengxwang@hotmail.com<br>> CC: petsc-users@mcs.anl.gov<br>> Subject: RE: [petsc-users] Scalability of PETSc on vesta.alcf<br>> Date: Mon, 20 Jan 2014 10:32:32 -0700<br>> <br>> Roc Wang <pengxwang@hotmail.com> writes:<br>> > I tried c16 for 1024 ranks and 2048 ranks, but the job cannot run<br>> > successfully. It seems the job was started but the program didn't<br>> > execute. Please take a look at the attached log file for 1024 with<br>> > c16 mode. Is this because some environment parameters I didn't set<br>> > right? Actually, the same program is only able to run with 1024<br>> > ranks in c1, c2 and c32, c64 modes and 2048 ranks in c64 mode.<br>> <br>> You have non-scalable "Generate Vector" and VecView (the latter maybe<br>> because you don't use MPI-IO?). It is probably failing at this step.<br>> <br>> | qsub -A SUGAR -t 00:10:00 -n 512 --proccount 2048 --mode script ./vesta.job<br>> <br>> I thought you said you were trying c16?<br><br>Yes, I said so. But, I tried both ways: qsub the executable and qsub script. The command is like this:<br><br>qsub -n 64 -t 10 --mode c16 -O p1024_c16 --env "F00=a:BAR=b" ./x.r -ksp_type bcgsl -ksp_bcgsl_ell 1 -sub_pc_type ilu -sub_pc_factor_levels 3 -sub_ksp_type preonly -my_ksp_monitor true -ksp_view -log_summary<br><br>the script:<br><br>#!/bin/bash<br><br>proN=1024<br><br>preName=p$proN<br><br>echo "Script JOB with Jobid COBALT_JOBID="$preName<br><br><br>qsub -A SUGAR -t 00:10:00 -n 64 --proccount $proN --mode script ./vesta.job<br><br><br>and vesta.job:<br><br>#!/bin/sh<br>Nrank=1024<br>echo Starting Cobalt job script<br><br>LOCARGS="--block $COBALT_PARTNAME ${COBALT_CORNER:+--corner} $COBALT_CORNER ${COBALT_SHAPE:+--shape} $COBALT_SHAPE"<br><br>runjob $LOCARGS -n $Nrank -p 16 : x.r -ksp_type bcgsl -ksp_bcgsl_ell 1 -sub_pc_type ilu -sub_pc_factor_levels 3 -sub_ksp_type preonly -my_ksp_monitor true -ksp_view -log_summary<br><br>echo End of jobscript.sh<br><br>exit 0<br><br>Both of them cannot run the program successfully. In these two ways, the runtime log showed the job started but no output to stdout file.<br><br>I just run the same program by:<br>qsub -n 16 -t 10 --mode c64 -O n1024_c64 --env "F00=a:BAR=b" ./x.r -ksp_type bcgsl -ksp_bcgsl_ell 1 -sub_pc_type ilu -sub_pc_factor_levels 3 -sub_ksp_type preonly -my_ksp_monitor true -ksp_view -log_summary<br><br>The job was able to run and the stdout file showed all the runtime output. If there is non-scalable "Generate Vector" and VecView (the latter maybe> because you don't use MPI-IO?), why is c64 mode able to run? It's sort of strange to me. Thanks.<br></div> </div></body>
</html>