<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I added a PETSc solver class to our particle-in-cell simulation
code and all calculations seem to be correct. However, some weak
scaling tests I did are rather disappointing because the solver's
runtime keeps increasing with system size although the number of
cores are scaled up accordingly. As a result, the solver's share
of the total runtime becomes more and more dominant and the system
sizes we aim for are unfeasible.</p>
<p>It's a simple 3D Poisson problem on a structured grid with
Dirichlet boundaries inside the domain, for which I found the
cg/gamg combo to work the fastest. Since KSPsolve() is called
during every timestep of the simulation to solve the same system
with a new rhs vector, assembling the matrix and other PETSc
objects should further not be a determining factor.</p>
<p>What puzzles me is that the convergence rate is actually good
(the residual decreases by an order of magnitude for every KSP
iteration) and the number of KSP iterations remains constant over
the course of a simulation and is equal for all tested systems.</p>
<p>I even increased the (fixed) system size per processor to 30^3
unknowns (which is significantly more than the recommended
10,000), but runtime is still not even close to being constant.</p>
<p>This leads me to the conclusion that either I configured PETSc
wrong, I don't call the correct PETSc-related functions, or
something goes terribly wrong with communication.</p>
<p>Could you have a look at the attached log_view files and tell me
if something is particularly odd? The system size per processor is
30^3 and the simulation ran over 1000 timesteps, which means
KSPsolve() was called equally often. I introduced two new logging
states - one for the first solve and the final setup and one for
the remaining solves.</p>
<p>The repeatedly called code segment is</p>
<blockquote><font size="-1">PetscScalar *b_array;<br>
VecGetArray(b, &b_array);</font><br>
<font size="-1">get_b(b_array);</font><br>
<font size="-1">VecRestoreArray(b, &barray);<br>
<br>
KSPSetTolerances(ksp,reltol,1E-50,1E5,1E4);<br>
<br>
PetscScalar *x_array;<br>
VecGetArray(x, &x_array);<br>
for (int i = 0; i < N_local; i++)<br>
x_array[i] = x_array_prev[i];<br>
VecRestoreArray(x, &x_array);<br>
<br>
KSPSolve(ksp,b,x);<br>
<br>
KSPGetSolution(ksp,&x);<br>
for (int i = 0; i < N_local; i++)<br>
x_array_prev[i] = x_array[i];<br>
</font><br>
<font size="-1">set_x(x_array);</font><br>
</blockquote>
<p>I noticed that for every individual KSP iteration, six vector
objects are created and destroyed (with CG, more with e.g. GMRES).
This seems kind of wasteful, is this supposed to be like this? Is
this even the reason for my problems? Apart from that, everything
seems quite normal to me (but I'm not the expert here).</p>
<p><br>
</p>
<p>Thanks in advance.</p>
<p>Michael<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</body>
</html>