<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Yes Matt, I am calling
VecAssemblyBegin/End() after VecSetValues.<br>
<br>
What brings me this idea is the ex30 exercice in
src/ksp/ksp/examples/tests/ex30.c<br>
where there was the read of a matrix, then VecSet, then
VecSetValues eventually<br>
(and it was working on GPU with mpiexec -np 2 ...). I played a lot
with it to discover<br>
VecSet was a turnaround to fix the crash of VecSetValues.<br>
<br>
I can send my modified ex30.c test to reproduce the bug.<br>
<br>
Pierre<br>
</div>
<blockquote
cite="mid:CAMYG4GnSVdgETO6ywcKGEzxVgJKMbneyRS8x1-59qU80cWopbg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Jul 29, 2014 at 9:58 AM,
Projet_TRIOU <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:triou@cea.fr" target="_blank">triou@cea.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hello Karl,<br>
<br>
For your interest, I understood why my code crashes on
GPU with np CPUs > 1.<br>
<br>
I am solving Ax=B with KSPSetInitialGuessNonzero set
to true so I need<br>
also to fill x with VecSetValues before each solve.<br>
<br>
VecSetValues works on GPU with np=1 but not for
np>1. I noticed that filling <br>
x with VecSet worked fine so when I create x, I now
initialize also with VecSet.<br>
(I suppose it copy/create something on GPU that
VecSetValues doesn't):<br>
<br>
Vec x;<br>
VecCreate(PETSC_COMM_WORLD,&x);<br>
VecSetSizes(x, nb_rows_, PETSC_DECIDE);<br>
VecSetType(x, VECMPICUSP);<br>
VecSet(x,0.0); // Needed on GPU during parallel
calculation, else it crashes<br>
<br>
// Later:<br>
for (int i=0; i<size; i++)<br>
{<br>
VecSetValues(b, 1, &colonne_globale,
&secmem(i), INSERT_VALUES);<br>
VecSetValues(x, 1, &colonne_globale,
&solution(i), INSERT_VALUES);<br>
colonne_globale++;<br>
}<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Are you calling VecAssemblyBegin/End() after this? It
is required for VecSetValues().</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div> Thanks for your help,<br>
<br>
Pierre<br>
</div>
<blockquote type="cite">
<div>Thanks Karl, and sorry for my mistake. Indeed,
ex9 runs in // with -pc_type none or -pc_type
jacobi.<br>
Good to know I have an example now with VecSetValues
I can try to replicate in my code.<br>
I will update the thread as soon as I return to some
more tests on it.<br>
<br>
Pierre<br>
</div>
<div>
<div class="h5">
<blockquote type="cite">Hi Pierre, <br>
<br>
I could reproduce the problem, but I could also
verify that the problem you ran into was due to
a currently unsupported block-Jacobi
preconditioner. That is, if you first comment
the KSPSolve for the ksp2 object in ex9,
recompile ex9, and then run <br>
<br>
$> mpiexec -np 2 ./ex9 -ksp_monitor -mat_type
aijcusp -vec_type cusp -pc_type none <br>
<br>
thinks should work out. This may also be
applicable to your case: Give it a try with
-pc_type none and let us whether the problem
with VecSetValues remains. (The use of no
preconditioner is certainly not great, but at
least it may give you a first result) <br>
<br>
Best regards, <br>
Karli <br>
<br>
<br>
<br>
On 03/26/2014 06:38 PM, Projet_TRIOU wrote: <br>
<blockquote type="cite">Hello all, <br>
<br>
My parallel code is really near to run using
GPU device <br>
thanks to PETSc but I am struggling with the
best way <br>
to fill the PETSc vectors in order in runs on
CPU & GPU. <br>
<br>
I was using in the past,
VecCreateMPIWithArray, to <br>
create vector but I had trouble with it on
GPU. <br>
<br>
So I followed the example in the
src/ksp/ksp/examples/tutorials/ex2.c <br>
and create now the vectors like this: <br>
<br>
// Build x <br>
ierr =
VecCreate(PETSC_COMM_WORLD,&SecondMembrePetsc_);
check(ierr); <br>
ierr = VecSetSizes(SecondMembrePetsc_,
nb_rows, nb_rows_tot); <br>
check(ierr); <br>
ierr =
VecSetFromOptions(SecondMembrePetsc_);
check(ierr); <br>
// Build b <br>
ierr =
VecDuplicate(SecondMembrePetsc_,&SolutionPetsc_);check(ierr);
<br>
<br>
And fills it with VecSetValues function. It
runs well on CPU and <br>
GPU but crashed only in parallel on GPU. It I
use VecSet instead of <br>
VecSetValues, it didn't crash (but of course
VecSet is not enough <br>
for me :-) <br>
<br>
I tried to find an example to reproduce for
you the problem, and I <br>
think src/ksp/ksp/examples/tutorials/ex9 (it
is usingVecSetValues,) <br>
is a good one. <br>
<br>
Or did I miss something (I also try
VecPlaceArray/VecRestoreArray <br>
but without success on GPU) ? <br>
<br>
Thanks, and yes you are right, "WARNING:
Using GPUs effectively is <br>
difficult!" :-) <br>
<br>
Pierre <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> ./ex9 -ksp_monitor <br>
0 KSP Residual norm 6.612932697792e+00 <br>
1 KSP Residual norm 4.261830032389e-01 <br>
2 KSP Residual norm 2.121746090851e-02 <br>
3 KSP Residual norm 1.233779841608e-03 <br>
4 KSP Residual norm 1.265903168531e-05 <br>
0 KSP Residual norm 1.309416176382e-05 <br>
0 KSP Residual norm 1.404919664063e-05 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> mpiexec -np 2 ./ex9 -ksp_monitor <br>
0 KSP Residual norm 2.496821857304e+02 <br>
1 KSP Residual norm 4.522206074831e+01 <br>
2 KSP Residual norm 1.959482408314e+01 <br>
3 KSP Residual norm 7.002013703407e+00 <br>
4 KSP Residual norm 2.144105201713e+00 <br>
5 KSP Residual norm 1.780095080270e-01 <br>
6 KSP Residual norm 5.642702243268e-02 <br>
7 KSP Residual norm 6.439343992306e-03 <br>
8 KSP Residual norm 3.012756374415e-04 <br>
Norm of error 0.000249108, Iterations 8 <br>
Norm of error 0.000715584, Iterations 6 <br>
0 KSP Residual norm 3.422287562824e-04 <br>
Norm of error 0.000249108, Iterations 0 <br>
Norm of error 0.000192805, Iterations 7 <br>
0 KSP Residual norm 4.140588954098e-04 <br>
Norm of error 0.000249108, Iterations 0 <br>
Norm of error 0.000109507, Iterations 7 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> ./ex9 -ksp_monitor -mat_type aijcusp
-vec_type cusp <br>
0 KSP Residual norm 6.612932697792e+00 <br>
1 KSP Residual norm 4.261830032389e-01 <br>
2 KSP Residual norm 2.121746090851e-02 <br>
3 KSP Residual norm 1.233779841608e-03 <br>
4 KSP Residual norm 1.265903168531e-05 <br>
0 KSP Residual norm 1.309416176403e-05 <br>
0 KSP Residual norm 1.404919664088e-05 <br>
<br>
sitre.intra.cea.fr:/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt/src/ksp/ksp/examples/tutorials
<br>
> mpiexec -np 2 ./ex9 -ksp_monitor
-mat_type aijcusp -vec_type cusp <br>
[0]PETSC ERROR: <br>
------------------------------------------------------------------------
<br>
[0]PETSC ERROR: Caught signal number 11 SEGV:
Segmentation Violation, <br>
probably memory access out of range <br>
[0]PETSC ERROR: Try option -start_in_debugger
or -on_error_attach_debugger <br>
[0]PETSC ERROR: or see <br>
<a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind"
target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>[0]PETSC
<br>
ERROR: or try <a moz-do-not-send="true"
href="http://valgrind.org" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to <br>
find memory corruption errors <br>
[0]PETSC ERROR: configure using
--with-debugging=yes, recompile, link, <br>
and run <br>
[0]PETSC ERROR: to get more information on the
crash. <br>
[0]PETSC ERROR: --------------------- Error
Message <br>
--------------------------------------------------------------
<br>
[0]PETSC ERROR: Signal received <br>
[0]PETSC ERROR: See <br>
<a moz-do-not-send="true"
href="http://http://www.mcs.anl.gov/petsc/documentation/faq.html"
target="_blank">http://http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble <br>
shooting. <br>
[0]PETSC ERROR: Petsc Development GIT
revision: v3.4.4-3713-g576f62e <br>
GIT Date: 2014-03-23 15:59:15 -0500 <br>
[0]PETSC ERROR: ./ex9 on a linux_opt named <a
moz-do-not-send="true"
href="http://sitre.intra.cea.fr"
target="_blank">sitre.intra.cea.fr</a> by
triou <br>
Wed Mar 26 18:32:34 2014 <br>
[0]PETSC ERROR: Configure options <br>
--prefix=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt
<br>
--with-single-library
--with-shared-libraries=0 --with-debugging=0 <br>
--with-errorchecking=1 --COPTFLAGS=" -O3
-fPIC " --CXXOPTFLAGS=" -O3 <br>
-fPIC " --FOPTFLAGS=" -O3 -fPIC "
--with-fortran=yes --with-clean=1 <br>
--download-scalapack=../scalapack-2.0.2.tgz <br>
--download-mumps=../MUMPS_4.10.0-p3.tar.gz
--download-superlu_dist=yes <br>
--download-parmetis=../parmetis-4.0.2-p4.tar.gz
<br>
--download-metis=../metis-5.0.2-p3.tar.gz <br>
--download-ptscotch=../ptscotch.tar.gz <br>
--download-hypre=../hypre-2.9.1a.tar.gz <br>
--with-valgrind-include=/work/triou/git/petsc-dev/Trio_U/exec/valgrind/include
<br>
--with-blas-lapack-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBLAPACK
--with-cuda=1 <br>
--with-cuda-dir=/work/triou/git/petsc-dev/Trio_U/exec/cuda5.5
<br>
--with-cusp=1 <br>
--with-cusp-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/cusplibrary-0.4.0
<br>
--with-thrust=1 --with-cuda-arch=sm_21
--with-ssl=0 <br>
--with-mpi-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBMPI/mpich
<br>
--with-x=1 <br>
[0]PETSC ERROR: #1 User provided function()
line 0 in unknown file <br>
[1]PETSC ERROR: <br>
------------------------------------------------------------------------
<br>
[1]PETSC ERROR: Caught signal number 11 SEGV:
Segmentation Violation, <br>
probably memory access out of range <br>
[1]PETSC ERROR: Try option -start_in_debugger
or -on_error_attach_debugger <br>
[1]PETSC ERROR: or see <br>
<a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind"
target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>[1]PETSC
<br>
ERROR: or try <a moz-do-not-send="true"
href="http://valgrind.org" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to <br>
find memory corruption errors <br>
[1]PETSC ERROR: configure using
--with-debugging=yes, recompile, link, <br>
and run <br>
[1]PETSC ERROR: application called
MPI_Abort(MPI_COMM_WORLD, 59) - process 0 <br>
to get more information on the crash. <br>
[1]PETSC ERROR: --------------------- Error
Message <br>
--------------------------------------------------------------
<br>
[1]PETSC ERROR: Signal received <br>
[1]PETSC ERROR: See <br>
<a moz-do-not-send="true"
href="http://http://www.mcs.anl.gov/petsc/documentation/faq.html"
target="_blank">http://http://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble <br>
shooting. <br>
[1]PETSC ERROR: Petsc Development GIT
revision: v3.4.4-3713-g576f62e <br>
GIT Date: 2014-03-23 15:59:15 -0500 <br>
[1]PETSC ERROR: ./ex9 on a linux_opt named <a
moz-do-not-send="true"
href="http://sitre.intra.cea.fr"
target="_blank">sitre.intra.cea.fr</a> by
triou <br>
Wed Mar 26 18:32:34 2014 <br>
[1]PETSC ERROR: Configure options <br>
--prefix=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/petsc/linux_opt
<br>
--with-single-library
--with-shared-libraries=0 --with-debugging=0 <br>
--with-errorchecking=1 --COPTFLAGS=" -O3
-fPIC " --CXXOPTFLAGS=" -O3 <br>
-fPIC " --FOPTFLAGS=" -O3 -fPIC "
--with-fortran=yes --with-clean=1 <br>
--download-scalapack=../scalapack-2.0.2.tgz <br>
--download-mumps=../MUMPS_4.10.0-p3.tar.gz
--download-superlu_dist=yes <br>
--download-parmetis=../parmetis-4.0.2-p4.tar.gz
<br>
--download-metis=../metis-5.0.2-p3.tar.gz <br>
--download-ptscotch=../ptscotch.tar.gz <br>
--download-hypre=../hypre-2.9.1a.tar.gz <br>
--with-valgrind-include=/work/triou/git/petsc-dev/Trio_U/exec/valgrind/include
<br>
--with-blas-lapack-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBLAPACK
--with-cuda=1 <br>
--with-cuda-dir=/work/triou/git/petsc-dev/Trio_U/exec/cuda5.5
<br>
--with-cusp=1 <br>
--with-cusp-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBPETSC/cusplibrary-0.4.0
<br>
--with-thrust=1 --with-cuda-arch=sm_21
--with-ssl=0 <br>
--with-mpi-dir=/work/triou/git/petsc-dev/Trio_U/lib/src/LIBMPI/mpich
<br>
--with-x=1 <br>
[1]PETSC ERROR: #1 User provided function()
line 0 in unknown file <br>
application called MPI_Abort(MPI_COMM_WORLD,
59) - process 1 <br>
<br>
<br>
<br>
-- <br>
*Trio_U support team* <br>
Marthe ROUX (01 69 08 00 02) Saclay <br>
Pierre LEDAC (04 38 78 91 49) Grenoble <br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</div>
</div>
<div>-- <br>
<b>Trio_U support team</b> <br>
<div class=""> Marthe ROUX (01 69 08 00 02) Saclay <br>
Pierre LEDAC (04 38 78 91 49) Grenoble </div>
</div>
<span class="HOEnZb"><font color="#888888"> </font></span></blockquote>
<span class="HOEnZb"><font color="#888888"> <br>
<br>
</font></span>
<div><span class="HOEnZb"><font color="#888888">-- <br>
<b>Trio_U support team</b> <br>
</font></span>
<div class=""> Marthe ROUX (01 69 08 00 02) Saclay <br>
Pierre LEDAC (04 38 78 91 49) Grenoble </div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener
</div>
</div>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<b>Trio_U support team</b>
<br>
Marthe ROUX (01 69 08 00 02) Saclay
<br>
Pierre LEDAC (04 38 78 91 49) Grenoble
</div>
</body>
</html>