<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Do you mean all the computational work are done on the GPU? <div><br><div>When I run ex5 with
-dm_vec_type veccusp -dm_mat_type mataijcusp, it appears the following error:</div><div><br></div><div><div>~/ex5\>./ex5 -dm_vec_type veccusp -dm_mat_type -log_summary ex5_log</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------------</div><div>[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type:</div><div>see http://www.mcs.anl.gov/petsc/documentation/installation.html#external!</div><div>[0]PETSC ERROR: Unknown vector type: veccusp!</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500</div><div>[0]PETSC ERROR: See docs/changes/index.html for recent updates.</div><div>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.</div><div>[0]PETSC ERROR: See docs/index.html for manual pages.</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: ./ex5 on a arch-cuda named hohhot by hongwang Mon Aug 6 12:27:19 2012</div><div>[0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib</div><div>[0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012</div><div>[0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: VecSetType() line 44 in src/vec/vec/interface/vecreg.c</div><div>[0]PETSC ERROR: DMCreateGlobalVector_DA() line 36 in src/dm/impls/da/dadist.c</div><div>[0]PETSC ERROR: DMCreateGlobalVector() line 443 in src/dm/interface/dm.c</div><div>[0]PETSC ERROR: DMDASetUniformCoordinates() line 58 in src/dm/impls/da/gr1.c</div><div>[0]PETSC ERROR: main() line 113 in src/snes/examples/tutorials/ex5.c</div><div>application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0</div><div>[unset]: aborting job:</div><div>application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0</div><div><br></div>Is there something wrong with the CUSP? My PETSc version is -dev, the cusp version I use is 0.3.1, CUDA version is 4.2.</div><div><br></div><div>Zeng Xiangze <br>在 2012-08-06 03:18:58，"Matthew Knepley" <knepley@gmail.com> 写道：<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">On Sun, Aug 5, 2012 at 10:24 AM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial"><div>Dear Matt,</div><div><br></div><div>Thank you for your suggestion. I'm learning to use the GPU effectively step by step. I think it's useful for the novice if there is a manual about using PETSc with CUDA.</div>
Each iteration is done, the VEC will be copied to the host to evaluate the stopping condition, is it right?</div></blockquote><div><br></div><div>No, if that was true, we would have given up long ago. My guess is that some of your Vecs are not the correct type.</div>
<div>Can you look at ex5 suing -dm_vec_type veccusp -dm_mat_type mataijcusp and mail <a href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a>?</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial"><div>Sincerely,</div><div>Zeng Xiangze<br><div><br></div><div>在 2012-08-05 20:27:55，"Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> 写道：<br>
<blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial">When I change the PC type to JACOBI, the KSP type to BICG, although the computational speed both in the GPU and CPU are higher than that when I use SOR+BCGS, the computational work in the GPU doesn't seem much more efficient, the speed only 20% higher. Is there any proposal? The attachments are the output of the log_summary.</div>
</blockquote><div><br></div><div>You also have to look at the log_summary:</div><div><br></div><div><div>VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0</div>
<div>VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0</div><div>MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div>
</div><div><br></div><div>1) I said to use GMRES for a reason. Listen to me. BiCG uses the transpose, which right now confuses the results</div><div><br></div><div>2) Look at the copies to/from the GPU. You should not be copying the vector 4000 times. Start simple until you understand</div>
<div> everything about how the code is running. Use -pc_type none -ksp_type gmres and see if you can understand the results.</div><div> Then try different KSP and PC. Trying everything at once does not help anyone, and it is not science.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;font-size:14px;font-family:arial"><div>Thank you!</div>
<div><br></div><div>Zeng Xiangze <br><br>At 2012-08-05 00:01:11,"Xiangze Zeng" <<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>> wrote:<br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
<div style="line-height:1.7;font-size:14px;font-family:arial">JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In <a href="http://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank">http://www.mcs.anl.gov/petsc/features/gpus.html</a>, I see "<span>
<span style="font-family:Simsun;line-height:normal;font-size:medium"></span><span><span style="line-height:normal;font-size:medium;font-family:Simsun">All of the Krylov methods except </span><code style="line-height:normal">KSPIBCGS</code><span style="line-height:normal;font-size:medium;font-family:Simsun"> run on the GPU.</span></span></span><span style="font-family:Simsun;line-height:normal;background-color:rgb(213,234,255);font-size:medium"></span> " I don't find KSPIBCGS in the manual, is it KSPBCGS?<br>
<br>在 2012-08-04 23:04:55，"Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> 写道：<br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial">Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message:<div><br><div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------------</div>
<div>[0]PETSC ERROR: Petsc has generated inconsistent data!</div><div>[0]PETSC ERROR: Divide by zero!</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500</div>
<div>[0]PETSC ERROR: See docs/changes/index.html for recent updates.</div><div>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.</div><div>[0]PETSC ERROR: See docs/index.html for manual pages.</div><div>
[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012</div><div>[0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib</div>
<div>[0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012</div><div>[0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20</div>
<div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c</div><div>[0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c</div>
<div>[0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c</div><div><br></div><div> And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. </div>
</div></div></div></blockquote><div><br></div><div>This is breakdown in that algorithm. Try GMRES.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial"><div><div><div>Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU?</div><br>在 2012-08-04 21:44:02，"Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> 写道：<br>
<blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial">After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary.</div>
</blockquote><div><br></div><div>1) Notice how the PCApply takes most of the time, so MatMult is not very important</div><div><br></div><div>2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU.</div>
<div> This means we do not support that PC on the GPU</div><div><br></div><div>There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP</div><div>work there, see <a href="http://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank">http://www.mcs.anl.gov/petsc/features/gpus.html</a>.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;font-size:14px;font-family:arial"><div><div>
At 2012-08-04 14:40:33,"Azamat Mametjanov" <<a href="mailto:azamat.mametjanov@gmail.com" target="_blank">azamat.mametjanov@gmail.com</a>> wrote:<br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
What happens if you try to re-run with "<span style="font-family:Monaco;font-size:11px">--with-debugging=no</span>"?<br><br><div class="gmail_quote">On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;font-size:14px;font-family:arial"><div>Dear Matt,</div><div><br></div>My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. <div>
<div>The size of the system is 2522469 x
2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. </div><div>The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. </div>
<div><br></div><div>Zeng Xiangze<br><div><br>在 2012-08-03 22:28:07，"Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> 写道：<div><div><br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng <span dir="ltr"><<a href="mailto:zengshixiangze@163.com" target="_blank">zengshixiangze@163.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial">Dear all,<div><br><div>When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter?</div>
<div></div></div></div></blockquote><div><br></div><div>For all performance questions, we at least need the output of -log_summary. However, we would also need to know</div><div><br></div><div> - The size and sparsity of your system</div>
<div><br></div><div> - The CPU and GPU you used (saying anything without knowing this is impossible)</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="line-height:1.7;font-size:14px;font-family:arial"><div><div>Thank you!</div><div><br></div><div>Sincerely,</div><div>Zeng Xiangze</div></div></div><br><br><span title="neteasefooter"><span></span></span></blockquote>
</div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>
</blockquote></div></div></div></div></div></div><br><br><span title="neteasefooter"><span></span></span></blockquote></div><br>
</blockquote><br><br></div></div><div>--<br><font face="arial, sans-serif" style="line-height:normal;font-size:small">Mailbox 379, School</font><span style="line-height:normal;font-size:13px;border-collapse:collapse"> of Physics</span><div style="line-height:normal;font-size:small">
<span style="font-size:13px;border-collapse:collapse">Shandong University</span></div><div style="line-height:normal;font-size:small"><span style="font-size:13px;border-collapse:collapse">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div>
</div></div><br><br><span title="neteasefooter"><span></span></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote><br><br><div>--<br><font face="arial, sans-serif" style="line-height:normal;font-size:small">Mailbox 379, School</font><span style="line-height:normal;font-size:13px;border-collapse:collapse"> of Physics</span><div style="line-height:normal;font-size:small">
<span style="font-size:13px;border-collapse:collapse">Shandong University</span></div><div style="line-height:normal;font-size:small"><span style="font-size:13px;border-collapse:collapse">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div>
</div></div></div></div><br><br><span title="neteasefooter"><span></span></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote><br><br><div>--<br><font face="arial, sans-serif" style="line-height:normal;font-size:small">Mailbox 379, School</font><span style="line-height:normal;font-size:13px;border-collapse:collapse"> of Physics</span><div style="line-height:normal;font-size:small">
<span style="font-size:13px;border-collapse:collapse">Shandong University</span></div><div style="line-height:normal;font-size:small"><span style="font-size:13px;border-collapse:collapse">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div>
</div></div><br><br><span title="neteasefooter"><span></span></span></blockquote><br><br><div>--<br><font face="arial, sans-serif" style="line-height:normal;font-size:small">Mailbox 379, School</font><span style="line-height:normal;font-size:13px;border-collapse:collapse"> of Physics</span><div style="line-height:normal;font-size:small">
<span style="font-size:13px;border-collapse:collapse">Shandong University</span></div><div style="line-height:normal;font-size:small"><span style="font-size:13px;border-collapse:collapse">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div>
</div></div></div><br><br><span title="neteasefooter"><span></span></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote><br><br><div>--<br><font face="arial, sans-serif" style="line-height:normal;font-size:small">Mailbox 379, School</font><span style="line-height:normal;font-size:13px;border-collapse:collapse"> of Physics</span><div style="line-height:normal;font-size:small">
<span style="font-size:13px;border-collapse:collapse">Shandong University</span></div><div style="line-height:normal;font-size:small"><span style="font-size:13px;border-collapse:collapse">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div>
</div></div></div></div><br><br><span title="neteasefooter"><span></span></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote><br><br><div>--<br><font face="arial, sans-serif" style="line-height: normal; font-size: small; ">Mailbox 379, School</font><span style="line-height: normal; font-size: 13px; border-collapse: collapse; "> of Physics</span><div style="line-height: normal; font-size: small; "><span style="font-size: 13px; border-collapse: collapse; ">Shandong University</span></div><div style="line-height: normal; font-size: small; "><span style="font-size: 13px; border-collapse: collapse; ">27 South Shanda Road, Jinan, Shandong, P.R.China, 250100</span></div></div></div></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>