<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Jul 29, 2014 at 9:58 AM, Projet_TRIOU <span dir="ltr"><<a href="mailto:triou@cea.fr" target="_blank">triou@cea.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hello Karl,<br>
<br>
For your interest, I understood why my code crashes on GPU with np
CPUs > 1.<br>
<br>
I am solving Ax=B with KSPSetInitialGuessNonzero set to true so I
need<br>
also to fill x with VecSetValues before each solve.<br>
<br>
VecSetValues works on GPU with np=1 but not for np>1. I noticed
that filling <br>
x with VecSet worked fine so when I create x, I now initialize
also with VecSet.<br>
(I suppose it copy/create something on GPU that VecSetValues
doesn't):<br>
<br>
Vec x;<br>
VecCreate(PETSC_COMM_WORLD,&x);<br>
VecSetSizes(x, nb_rows_, PETSC_DECIDE);<br>
VecSetType(x, VECMPICUSP);<br>
VecSet(x,0.0); // Needed on GPU during parallel calculation, else
it crashes<br>
<br>
// Later:<br>
for (int i=0; i<size; i++)<br>
{<br>
VecSetValues(b, 1, &colonne_globale, &secmem(i),
INSERT_VALUES);<br>
VecSetValues(x, 1, &colonne_globale, &solution(i),
INSERT_VALUES);<br>
colonne_globale++;<br>
}<br></div></div></blockquote><div><br></div><div>Are you calling VecAssemblyBegin/End() after this? It is required for VecSetValues().</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><div>
Thanks for your help,<br>
<br>
Pierre<br>
</div>
<blockquote type="cite">
<div>Thanks Karl, and sorry for my
mistake. Indeed, ex9 runs in // with -pc_type none or -pc_type
jacobi.<br>
Good to know I have an example now with VecSetValues I can try
to replicate in my code.<br>
I will update the thread as soon as I return to some more tests
on it.<br>
<br>
Pierre<br>
</div><div><div class="h5">
<blockquote type="cite">Hi Pierre, <br>
<br>
I could reproduce the problem, but I could also verify that the
problem you ran into was due to a currently unsupported
block-Jacobi preconditioner. That is, if you first comment the
KSPSolve for the ksp2 object in ex9, recompile ex9, and then run
<br>
<br>
$> mpiexec -np 2 ./ex9 -ksp_monitor -mat_type aijcusp
-vec_type cusp -pc_type none <br>
<br>
thinks should work out. This may also be applicable to your
case: Give it a try with -pc_type none and let us whether the
problem with VecSetValues remains. (The use of no preconditioner
is certainly not great, but at least it may give you a first
result) <br>
<br>
Best regards, <br>
Karli <br>
<br>
<br>
<br>
On 03/26/2014 06:38 PM, Projet_TRIOU wrote: <br>
<blockquote type="cite">Hello all, <br>
<br>
My parallel code is really near to run using GPU device <br>
thanks to PETSc but I am struggling with the best way <br>
to fill the PETSc vectors in order in runs on CPU & GPU. <br>
<br>
I was using in the past, VecCreateMPIWithArray, to <br>
create vector but I had trouble with it on GPU. <br>
<br>
So I followed the example in the
src/ksp/ksp/examples/tutorials/ex2.c <br>
and create now the vectors like this: <br>
<br>
// Build x <br>
ierr = VecCreate(PETSC_COMM_WORLD,&SecondMembrePetsc_);
check(ierr); <br>
ierr = VecSetSizes(SecondMembrePetsc_, nb_rows,
nb_rows_tot); <br>
check(ierr); <br>
ierr = VecSetFromOptions(SecondMembrePetsc_); check(ierr);
<br>
// Build b <br>
ierr =
VecDuplicate(SecondMembrePetsc_,&SolutionPetsc_);check(ierr);
<br>
<br>
And fills it with VecSetValues function. It runs well on CPU
and <br>
GPU but crashed only in parallel on GPU. It I use VecSet
instead of <br>
VecSetValues, it didn't crash (but of course VecSet is not
enough <br>
for me :-) <br>
<br>
I tried to find an example to reproduce for you the problem,
and I <br>
think src/ksp/ksp/examples/tutorials/ex9 (it is
usingVecSetValues,) <br>
is a good one. <br>
<br>
Or did I miss something (I also try
VecPlaceArray/VecRestoreArray <br>
but without success on GPU) ? <br>
<br>
Thanks, and yes you are right, "WARNING: Using GPUs
effectively is <br>
difficult!" :-) <br>
<br>
Pierre <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> ./ex9 -ksp_monitor <br>
0 KSP Residual norm 6.612932697792e+00 <br>
1 KSP Residual norm 4.261830032389e-01 <br>
2 KSP Residual norm 2.121746090851e-02 <br>
3 KSP Residual norm 1.233779841608e-03 <br>
4 KSP Residual norm 1.265903168531e-05 <br>
0 KSP Residual norm 1.309416176382e-05 <br>
0 KSP Residual norm 1.404919664063e-05 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> mpiexec -np 2 ./ex9 -ksp_monitor <br>
0 KSP Residual norm 2.496821857304e+02 <br>
1 KSP Residual norm 4.522206074831e+01 <br>
2 KSP Residual norm 1.959482408314e+01 <br>
3 KSP Residual norm 7.002013703407e+00 <br>
4 KSP Residual norm 2.144105201713e+00 <br>
5 KSP Residual norm 1.780095080270e-01 <br>
6 KSP Residual norm 5.642702243268e-02 <br>
7 KSP Residual norm 6.439343992306e-03 <br>
8 KSP Residual norm 3.012756374415e-04 <br>
Norm of error 0.000249108, Iterations 8 <br>
Norm of error 0.000715584, Iterations 6 <br>
0 KSP Residual norm 3.422287562824e-04 <br>
Norm of error 0.000249108, Iterations 0 <br>
Norm of error 0.000192805, Iterations 7 <br>
0 KSP Residual norm 4.140588954098e-04 <br>
Norm of error 0.000249108, Iterations 0 <br>
Norm of error 0.000109507, Iterations 7 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> ./ex9 -ksp_monitor -mat_type aijcusp -vec_type cusp <br>
0 KSP Residual norm 6.612932697792e+00 <br>
1 KSP Residual norm 4.261830032389e-01 <br>
2 KSP Residual norm 2.121746090851e-02 <br>
3 KSP Residual norm 1.233779841608e-03 <br>
4 KSP Residual norm 1.265903168531e-05 <br>
0 KSP Residual norm 1.309416176403e-05 <br>
0 KSP Residual norm 1.404919664088e-05 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> mpiexec -np 2 ./ex9 -ksp_monitor -mat_type aijcusp
-vec_type cusp <br>
[0]PETSC ERROR: <br>
------------------------------------------------------------------------
<br>
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
Violation, <br>
probably memory access out of range <br>
[0]PETSC ERROR: Try option -start_in_debugger or
-on_error_attach_debugger <br>
[0]PETSC ERROR: or see <br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>[0]PETSC
<br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to <br>
find memory corruption errors <br>
[0]PETSC ERROR: configure using --with-debugging=yes,
recompile, link, <br>
and run <br>
[0]PETSC ERROR: to get more information on the crash. <br>
[0]PETSC ERROR: --------------------- Error Message <br>
--------------------------------------------------------------
<br>
[0]PETSC ERROR: Signal received <br>
[0]PETSC ERROR: See <br>
<a href="http://http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble <br>
shooting. <br>
[0]PETSC ERROR: Petsc Development GIT revision:
v3.4.4-3713-g576f62e <br>
GIT Date: 2014-03-23 15:59:15 -0500 <br>
[0]PETSC ERROR: ./ex9 on a linux_opt named <a href="http://sitre.intra.cea.fr" target="_blank">sitre.intra.cea.fr</a>
by triou <br>
Wed Mar 26 18:32:34 2014 <br>
[0]PETSC ERROR: Configure options <br>
--prefix=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt
<br>
--with-single-library --with-shared-libraries=0
--with-debugging=0 <br>
--with-errorchecking=1 --COPTFLAGS=" -O3 -fPIC "
--CXXOPTFLAGS=" -O3 <br>
-fPIC " --FOPTFLAGS=" -O3 -fPIC " --with-fortran=yes
--with-clean=1 <br>
--download-scalapack=../scalapack-2.0.2.tgz <br>
--download-mumps=../MUMPS_4.10.0-p3.tar.gz
--download-superlu_dist=yes <br>
--download-parmetis=../parmetis-4.0.2-p4.tar.gz <br>
--download-metis=../metis-5.0.2-p3.tar.gz <br>
--download-ptscotch=../ptscotch.tar.gz <br>
--download-hypre=../hypre-2.9.1a.tar.gz <br>
--with-valgrind-include=/work/triou/git/petsc-dev/Trio_U/exec/valgrind/include
<br>
--with-blas-lapack-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBLAPACK
--with-cuda=1 <br>
--with-cuda-dir=/work/triou/git/petsc-dev/Trio_U/exec/cuda5.5
<br>
--with-cusp=1 <br>
--with-cusp-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/cusplibrary-0.4.0
<br>
--with-thrust=1 --with-cuda-arch=sm_21 --with-ssl=0 <br>
--with-mpi-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBMPI/mpich
<br>
--with-x=1 <br>
[0]PETSC ERROR: #1 User provided function() line 0 in unknown
file <br>
[1]PETSC ERROR: <br>
------------------------------------------------------------------------
<br>
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
Violation, <br>
probably memory access out of range <br>
[1]PETSC ERROR: Try option -start_in_debugger or
-on_error_attach_debugger <br>
[1]PETSC ERROR: or see <br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>[1]PETSC
<br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to <br>
find memory corruption errors <br>
[1]PETSC ERROR: configure using --with-debugging=yes,
recompile, link, <br>
and run <br>
[1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD,
59) - process 0 <br>
to get more information on the crash. <br>
[1]PETSC ERROR: --------------------- Error Message <br>
--------------------------------------------------------------
<br>
[1]PETSC ERROR: Signal received <br>
[1]PETSC ERROR: See <br>
<a href="http://http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble <br>
shooting. <br>
[1]PETSC ERROR: Petsc Development GIT revision:
v3.4.4-3713-g576f62e <br>
GIT Date: 2014-03-23 15:59:15 -0500 <br>
[1]PETSC ERROR: ./ex9 on a linux_opt named <a href="http://sitre.intra.cea.fr" target="_blank">sitre.intra.cea.fr</a>
by triou <br>
Wed Mar 26 18:32:34 2014 <br>
[1]PETSC ERROR: Configure options <br>
--prefix=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt
<br>
--with-single-library --with-shared-libraries=0
--with-debugging=0 <br>
--with-errorchecking=1 --COPTFLAGS=" -O3 -fPIC "
--CXXOPTFLAGS=" -O3 <br>
-fPIC " --FOPTFLAGS=" -O3 -fPIC " --with-fortran=yes
--with-clean=1 <br>
--download-scalapack=../scalapack-2.0.2.tgz <br>
--download-mumps=../MUMPS_4.10.0-p3.tar.gz
--download-superlu_dist=yes <br>
--download-parmetis=../parmetis-4.0.2-p4.tar.gz <br>
--download-metis=../metis-5.0.2-p3.tar.gz <br>
--download-ptscotch=../ptscotch.tar.gz <br>
--download-hypre=../hypre-2.9.1a.tar.gz <br>
--with-valgrind-include=/work/triou/git/petsc-dev/Trio_U/exec/valgrind/include
<br>
--with-blas-lapack-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBLAPACK
--with-cuda=1 <br>
--with-cuda-dir=/work/triou/git/petsc-dev/Trio_U/exec/cuda5.5
<br>
--with-cusp=1 <br>
--with-cusp-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/cusplibrary-0.4.0
<br>
--with-thrust=1 --with-cuda-arch=sm_21 --with-ssl=0 <br>
--with-mpi-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBMPI/mpich
<br>
--with-x=1 <br>
[1]PETSC ERROR: #1 User provided function() line 0 in unknown
file <br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 <br>
<br>
<br>
<br>
-- <br>
*Trio_U support team* <br>
Marthe ROUX (01 69 08 00 02) Saclay <br>
Pierre LEDAC (04 38 78 91 49) Grenoble <br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</div></div><div>-- <br>
<b>Trio_U support team</b> <br><div class="">
Marthe ROUX (01 69 08 00 02) Saclay <br>
Pierre LEDAC (04 38 78 91 49) Grenoble </div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
</font></span><div><span class="HOEnZb"><font color="#888888">-- <br>
<b>Trio_U support team</b>
<br></font></span><div class="">
Marthe ROUX (01 69 08 00 02) Saclay
<br>
Pierre LEDAC (04 38 78 91 49) Grenoble
</div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div>