[petsc-users] Is OpenMP still available for PETSc?

[Barry Smith]

   You may need to ask a slurm expert. I have no idea what cpus-per-task means


> On Jul 4, 2017, at 4:16 AM, Damian Kaliszan <damian at man.poznan.pl> wrote:
> 
> Hi,
> 
> Yes, this is exactly what I meant.
> Please find attached output for 2 input datasets and for 2 various slurm configs each:
> 
> A/ Matrix size=8000000x8000000
> 
> 1/ slurm-14432809.out,   930 ksp steps, ~90 secs
> 
> 
> #SBATCH --nodes=2
> #SBATCH --ntasks=32
> #SBATCH --ntasks-per-node=16
> #SBATCH --cpus-per-task=4
> 
> 2/  slurm-14432810.out , 100.000 ksp steps, ~9700 secs
> 
> #SBATCH --nodes=2
> #SBATCH --ntasks=32
> #SBATCH --ntasks-per-node=16
> #SBATCH --cpus-per-task=2
> 
> 
> 
> B/ Matrix size=1000x1000
> 
> 1/  slurm-23716.out, 511 ksp steps, ~ 28 secs
> #SBATCH --nodes=1
> #SBATCH --ntasks=64
> #SBATCH --ntasks-per-node=64
> #SBATCH --cpus-per-task=4
> 
> 
> 2/ slurm-23718.out, 94 ksp steps, ~ 4 secs
> 
> #SBATCH --nodes=1
> #SBATCH --ntasks=4
> #SBATCH --ntasks-per-node=4
> #SBATCH --cpus-per-task=4
> 
> 
> I would really appreciate any help...:)
> 
> Best,
> Damian
> 
> 
> 
> W liście datowanym 3 lipca 2017 (16:29:15) napisano:
> 
> 
> On Mon, Jul 3, 2017 at 9:23 AM, Damian Kaliszan <damian at man.poznan.pl> wrote:
> Hi,
> 
> 
> >> 1) You can call Bcast on PETSC_COMM_WORLD
> 
> To  be  honest  I  can't find Bcast method in petsc4py.PETSc.Comm (I'm
> using petsc4py)
> 
> >> 2) If you are using WORLD, the number of iterates will be the same on each process since iteration is collective.
> 
> Yes,  this  is  how  it  should  be.  But  what  I noticed is that for
> different  --cpus-per-task  numbers  in  slurm  script I get different
> number  of  solver iterations which is in turn related to timings. The
> imparity    is    huge.  For  example  for   some configurations where  --cpus-per-task=1 I receive 900
> iterations and  for --cpus-per-task=2 I receive valid number of 100.000 which is set as max
> iter number set when setting solver tolerances.
> 
> I am trying to understand what you are saying. You mean that you make 2 different runs and get a different
> number of iterates with a KSP? In order to answer questions about convergence, we need to see the output
> of
> 
> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
> 
> for all cases.
> 
> Thanks,
> 
>    Matt
> 
> Best,
> Damian
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> http://www.caam.rice.edu/~mk51/
> 
> 
> 
> 
> 
> -------------------------------------------------------
> Damian Kaliszan
> 
> Poznan Supercomputing and Networking Center
> HPC and Data Centres Technologies
> ul. Jana Pawła II 10
> 61-139 Poznan
> POLAND
> 
> phone (+48 61) 858 5109
> e-mail damian at man.poznan.pl
> www - http://www.man.poznan.pl/
> -------------------------------------------------------
> <slum_output.zip>