<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Dave,<br>
<br>
I re-run the test using bjacobi as the preconditioner on the coarse
mesh of telescope. The Grid is 3072*256*768 and process mesh is
96*8*24. The petsc option file is attached.<br>
I still got the "Out Of Memory" error. The error occurred before the
linear solver finished one step. So I don't have the full info from
ksp_view. The info from ksp_view_pre is attached. <br>
It seems to me that the error occurred when the decomposition was
going to be changed.<br>
I had another test with a grid of 1536*128*384 and the same process
mesh as above. There was no error. The ksp_view info is attached for
comparison.<br>
Thank you.<br>
<br>
Frank<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 07/08/2016 10:38 PM, Dave May wrote:<br>
</div>
<blockquote
cite="mid:CAJ98EDqfz2hr1s8SoLCwPceqbB+bsZp1ryibGngRn+kNTDrocA@mail.gmail.com"
type="cite"><br>
<br>
On Saturday, 9 July 2016, frank <<a moz-do-not-send="true"
href="javascript:_e(%7B%7D,'cvml','hengjiew@uci.edu');"
target="_blank">hengjiew@uci.edu</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Barry and
Dave,<br>
<br>
Thank both of you for the advice.<br>
<br>
@Barry<br>
I made a mistake in the file names in last email. I attached the
correct files this time.<br>
For all the three tests, 'Telescope' is used as the coarse
preconditioner.<br>
<br>
== Test1: Grid: 1536*128*384, Process Mesh: 48*4*12<br>
Part of the memory usage: Vector 125 124 3971904
0.<br>
Matrix 101 101
9462372 0<br>
<br>
== Test2: Grid: 1536*128*384, Process Mesh: 96*8*24<br>
Part of the memory usage: Vector 125 124 681672
0.<br>
Matrix 101 101
1462180 0.<br>
<br>
In theory, the memory usage in Test1 should be 8 times of Test2.
In my case, it is about 6 times.<br>
<br>
== Test3: Grid: 3072*256*768, Process Mesh: 96*8*24.
Sub-domain per process: 32*32*32<br>
Here I get the out of memory error.<br>
<br>
I tried to use -mg_coarse jacobi. In this way, I don't need to
set -mg_coarse_ksp_type and -mg_coarse_pc_type explicitly,
right?<br>
The linear solver didn't work in this case. Petsc output some
errors.<br>
<br>
@Dave<br>
In test3, I use only one instance of 'Telescope'. On the coarse
mesh of 'Telescope', I used LU as the preconditioner instead of
SVD.<br>
If my set the levels correctly, then on the last coarse mesh of
MG where it calls 'Telescope', the sub-domain per process is
2*2*2.<br>
On the last coarse mesh of 'Telescope', there is only one grid
point per process.<br>
I still got the OOM error. The detailed petsc option file is
attached.</blockquote>
<div><br>
</div>
<div>Do you understand the expected memory usage for the
particular parallel LU implementation you are using? I don't
(seriously). Replace LU with bjacobi and re-run this test. My
point about solver debugging is still valid. </div>
<div><br>
</div>
<div>And please send the result of KSPView so we can see what is
actually used in the computations</div>
<div><br>
</div>
<div>Thanks</div>
<div> Dave</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Thank you so much.<br>
<br>
Frank<br>
<br>
<br>
<br>
On 07/06/2016 02:51 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Jul 6, 2016, at 4:19 PM, frank <<a
moz-do-not-send="true"><a class="moz-txt-link-abbreviated" href="mailto:hengjiew@uci.edu">hengjiew@uci.edu</a></a>> wrote:<br>
<br>
Hi Barry,<br>
<br>
Thank you for you advice.<br>
I tried three test. In the 1st test, the grid is
3072*256*768 and the process mesh is 96*8*24.<br>
The linear solver is 'cg' the preconditioner is 'mg' and
'telescope' is used as the preconditioner at the coarse
mesh.<br>
The system gives me the "Out of Memory" error before the
linear system is completely solved.<br>
The info from '-ksp_view_pre' is attached. I seems to me
that the error occurs when it reaches the coarse mesh.<br>
<br>
The 2nd test uses a grid of 1536*128*384 and process mesh is
96*8*24. The 3rd test uses the same grid but a different
process mesh 48*4*12.<br>
</blockquote>
Are you sure this is right? The total matrix and vector
memory usage goes from 2nd test<br>
Vector 384 383 8,193,712
0.<br>
Matrix 103 103 11,508,688
0.<br>
to 3rd test<br>
Vector 384 383 1,590,520
0.<br>
Matrix 103 103 3,508,664
0.<br>
that is the memory usage got smaller but if you have only
1/8th the processes and the same grid it should have gotten
about 8 times bigger. Did you maybe cut the grid by a factor
of 8 also? If so that still doesn't explain it because the
memory usage changed by a factor of 5 something for the
vectors and 3 something for the matrices.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
The linear solver and petsc options in 2nd and 3rd tests are
the same in 1st test. The linear solver works fine in both
test.<br>
I attached the memory usage of the 2nd and 3rd tests. The
memory info is from the option '-log_summary'. I tried to
use '-momery_info' as you suggested, but in my case petsc
treated it as an unused option. It output nothing about the
memory. Do I need to add sth to my code so I can use
'-memory_info'?<br>
</blockquote>
Sorry, my mistake the option is -memory_view<br>
<br>
Can you run the one case with -memory_view and -mg_coarse
jacobi -ksp_max_it 1 (just so it doesn't iterate forever) to
see how much memory is used without the telescope? Also run
case 2 the same way.<br>
<br>
Barry<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
In both tests the memory usage is not large.<br>
<br>
It seems to me that it might be the 'telescope'
preconditioner that allocated a lot of memory and caused the
error in the 1st test.<br>
Is there is a way to show how much memory it allocated?<br>
<br>
Frank<br>
<br>
On 07/05/2016 03:37 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Frank,<br>
<br>
You can run with -ksp_view_pre to have it "view" the
KSP before the solve so hopefully it gets that far.<br>
<br>
Please run the problem that does fit with
-memory_info when the problem completes it will show the
"high water mark" for PETSc allocated memory and total
memory used. We first want to look at these numbers to see
if it is using more memory than you expect. You could also
run with say half the grid spacing to see how the memory
usage scaled with the increase in grid points. Make the
runs also with -log_view and send all the output from
these options.<br>
<br>
Barry<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Jul 5, 2016, at 5:23 PM, frank <<a
moz-do-not-send="true"><a class="moz-txt-link-abbreviated" href="mailto:hengjiew@uci.edu">hengjiew@uci.edu</a></a>> wrote:<br>
<br>
Hi,<br>
<br>
I am using the CG ksp solver and Multigrid
preconditioner to solve a linear system in parallel.<br>
I chose to use the 'Telescope' as the preconditioner on
the coarse mesh for its good performance.<br>
The petsc options file is attached.<br>
<br>
The domain is a 3d box.<br>
It works well when the grid is 1536*128*384 and the
process mesh is 96*8*24. When I double the size of grid
and keep the same process mesh and petsc options, I get
an "out of memory" error from the super-cluster I am
using.<br>
Each process has access to at least 8G memory, which
should be more than enough for my application. I am sure
that all the other parts of my code( except the linear
solver ) do not use much memory. So I doubt if there is
something wrong with the linear solver.<br>
The error occurs before the linear system is completely
solved so I don't have the info from ksp view. I am not
able to re-produce the error with a smaller problem
either.<br>
In addition, I tried to use the block jacobi as the
preconditioner with the same grid and same
decomposition. The linear solver runs extremely slow but
there is no memory error.<br>
<br>
How can I diagnose what exactly cause the error?<br>
Thank you so much.<br>
<br>
Frank<br>
<petsc_options.txt><br>
</blockquote>
</blockquote>
<ksp_view_pre.txt><memory_test2.txt><memory_test3.txt><petsc_options.txt><br>
</blockquote>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</body>
</html>