<div dir="ltr">Hi Karl,<div><br></div><div>Thanks! It worked. However the following cases still don't work:</div><div><br></div><div>1) Using ComputeResidual (not ComputeResidualViennaCL) but with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The SNES solver converges but gives nonsensical results. I just get a flashing blob.</div>
<div><br></div><div>2) Using ComputeResidualViennaCL (not ComputeResidual) and <b>without</b> DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The SNES solver converges but gives nonsensical results. I just get a flashing blob as 1) above. This case probably should not work anyway cause I'm not sure if VecViennaCLGetArray works if the Vecs have not been set to ViennaCL.</div>
<div><br></div><div>Cheers,</div><div>Mani</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Jan 26, 2014 at 4:25 AM, Karl Rupp <span dir="ltr"><<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mani,<br>
<br>
I tested your code with branch `karlrupp/viennacl-getarray`, which is also in `next` and just got merged to `master`.<br>
<br>
Initially I had to fix up a missing double precision pragma for OpenCL, two GCC warnings and a missing local work group size specification in your code to get it to run at all, but then I immediately got the correct results. The fixed code is attached, changes are tagged with '//FIXED'.<br>
<br>
Except for possibly machine-specific issues, the problem was the missing local work size specification in your code. Always specify both the local and the global work sizes and make sure that the global sizes are divisible by the respective local work sizes. Good local work sizes are usually in the range 64-256 workers total, hence I set the two local work size dimensions to 8. Also, I recommend some additional checks for 'REAL' to be identical to 'PetscScalar', otherwise you will get garbage because of obvious float<->double incompatibilities at the binary level.<br>
<br>
Best regards,<br>
Karli<div class="im"><br>
<br>
<br>
<br>
On 01/25/2014 10:22 PM, Mani Chandra wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Karl,<br>
<br><div class="im">
Thanks for looking into it. Do let me know if there is anything I can do<br>
to help you debug this. Attached is the code.<br>
<br>
Cheers,<br>
Mani<br>
<br>
<br>
On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a><br></div><div class="im">
<mailto:<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>>> wrote:<br>
<br>
Hi Mani,<br>
<br></div><div class="im">
could you please send me the code including the error checks?<br>
<br>
Thanks and best regards,<br>
Karli<br>
<br>
<br>
<br>
On 01/25/2014 10:07 PM, Mani Chandra wrote:<br>
<br>
Hi Karl,<br>
<br>
I now checked the error flags of all petsc functions in all<br>
functions. I<br>
also recompiled petsc with debug. Here's the report:<br>
<br>
Case 1) Still works. All good.<br>
<br>
Case 2) This is the case with ComputeResidual using VecGetArrays but<br>
with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,<br>
MATAIJVIENNACL).<br>
The flags don't show any errors. *The programs proceeds smoothly<br>
but the<br>
solution is wrong even though SNES is converging.*<br>
<br>
<br>
Case 3) This is the case with ComputeResidualViennaCL using<br>
VecViennaCLGetArrays with and without DMSetVecType(da,<br>
VECVIENNACL) and<br>
DMSetMatType(da, MATAIJVIENNACL). In both cases I get the<br>
following error:<br>
<br>
[0]PETSC ERROR: TSComputeIFunction() line 676 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/interface/ts.c<div class="im"><br>
[0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/impls/implicit/theta/<u></u>theta.__c<div class="im"><br>
[0]PETSC ERROR: SNESTSFormFunction() line 3499 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/interface/ts.c<div class="im"><br>
[0]PETSC ERROR: SNESComputeFunction() line 2089 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_snes/interface/snes.c<div class="im"><br>
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_snes/impls/ls/ls.c<div class="im"><br>
[0]PETSC ERROR: SNESSolve() line 3812 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_snes/interface/snes.c<div class="im"><br>
[0]PETSC ERROR: TSStep_Theta() line 183 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/impls/implicit/theta/<u></u>theta.__c<div class="im"><br>
[0]PETSC ERROR: TSStep() line 2625 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/interface/ts.c<div class="im"><br>
[0]PETSC ERROR: TSSolve() line 2741 in<br></div>
/home/mc/Downloads/petsc/src/_<u></u>_ts/interface/ts.c<div class="im"><br>
[0]PETSC ERROR: main() line 83 in<br></div>
/home/mc/PhD/opencl_tests/__<u></u>petsc_opencl/petsc_opencl.cpp<br>
------------------------------<u></u>__----------------------------<u></u>--__--------------<div class="im"><br>
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>
with errorcode -473550369.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br></div>
------------------------------<u></u>__----------------------------<u></u>--__--------------<div class="im"><br>
<br>
The error occured at VecViennaCLGetArrayRead/Write in my<br>
ComputeResidualViennaCL function. Note that with the petsc debug<br>
mode,<br>
the code crashes with the above error even if I don't catch the<br>
error<br>
codes. Might be because of some VecViennaCLGetArray calls inside<br>
petsc.<br>
<br>
Cheers,<br>
Mani<br>
<br>
<br>
On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a><br>
<mailto:<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>><br></div><div><div class="h5">
<mailto:<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a> <mailto:<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>>>> wrote:<br>
<br>
Hi Mani,<br>
<br>
please check the return value of *all* function calls from<br>
PETSc, e.g.<br>
ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);<br>
instead of just<br>
DMCreateGlobalVector(da, &soln);<br>
Most likely one of the routines threw an error, but your<br>
code just<br>
kept going, producing wrong results.<br>
<br>
Best regards,<br>
Karli<br>
<br>
<br>
<br>
On 01/25/2014 05:35 AM, Mani Chandra wrote:<br>
<br>
Hi Everyone,<br>
<br>
I'm trying to use TS with ViennaCL vecs/mats and residual<br>
evaluation on<br>
device and have encountered some problems. I have<br>
attached a<br>
small test<br>
code that illustrates the issue.<br>
<br>
The code simply advects a blob diagonally using TS. I have<br>
written the<br>
residual evaluation function using 1) the usual Petsc<br>
vectors<br>
(VecGetArray) and 2) using ViennaCL vectors<br></div></div>
(VecViennaCLGetArrayRead/____<u></u>Write).<div><div class="h5"><br>
<br>
<br>
Run the code using the following:<br>
./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps<br>
1000 -ts_type<br>
theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000<br>
-ts_monitor_draw_solution<br>
<br>
Case 1) No ViennaCL anywhere. I simply use the usual Petsc<br>
vectors and<br>
set the residual evaluation function as ComputeResidual<br>
(line<br>
no. 55).<br>
This case works and the blob is indeed advected as can<br>
be seen. (I<br>
haven't bothered with the boundaries. The simulation<br>
just stops<br>
before<br>
the blob hits the boundaries).<br>
<br>
Case 2) We again use the ComputeResidual but now enable<br>
ViennaCL<br>
vecs<br>
and mats (line nos. 48, 49). This case does NOT work.<br>
The SNES<br>
monitor<br>
shows convergence but the solution makes no sense.<br>
<br>
Case 3) We now use ComputeResidualViennaCL (line no.<br>
56). This<br>
does NOT<br>
work either with or without enabling the ViennaCL vecs<br>
(line<br>
nos. 48, 49).<br>
<br>
Cheers,<br>
Mani<br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote>
<br>
</blockquote></div><br></div>