<div dir="ltr">Hi Karl,<br><br>Thanks for looking into it. Do let me know if there is anything I can do to help you debug this. Attached is the code.<br><br>Cheers,<br>Mani<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <span dir="ltr"><<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Mani,<br>
<br>
could you please send me the code including the error checks?<br>
<br>
Thanks and best regards,<br>
Karli<div class="im"><br>
<br>
<br>
On 01/25/2014 10:07 PM, Mani Chandra wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Hi Karl,<br>
<br>
I now checked the error flags of all petsc functions in all functions. I<br>
also recompiled petsc with debug. Here's the report:<br>
<br>
Case 1) Still works. All good.<br>
<br>
Case 2) This is the case with ComputeResidual using VecGetArrays but<br>
with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL).<br></div>
The flags don't show any errors. *The programs proceeds smoothly but the<br>
solution is wrong even though SNES is converging.*<div><div class="h5"><br>
<br>
Case 3) This is the case with ComputeResidualViennaCL using<br>
VecViennaCLGetArrays with and without DMSetVecType(da, VECVIENNACL) and<br>
DMSetMatType(da, MATAIJVIENNACL). In both cases I get the following error:<br>
<br>
[0]PETSC ERROR: TSComputeIFunction() line 676 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/interface/ts.c<br>
[0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/impls/implicit/theta/theta.<u></u>c<br>
[0]PETSC ERROR: SNESTSFormFunction() line 3499 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/interface/ts.c<br>
[0]PETSC ERROR: SNESComputeFunction() line 2089 in<br>
/home/mc/Downloads/petsc/src/<u></u>snes/interface/snes.c<br>
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in<br>
/home/mc/Downloads/petsc/src/<u></u>snes/impls/ls/ls.c<br>
[0]PETSC ERROR: SNESSolve() line 3812 in<br>
/home/mc/Downloads/petsc/src/<u></u>snes/interface/snes.c<br>
[0]PETSC ERROR: TSStep_Theta() line 183 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/impls/implicit/theta/theta.<u></u>c<br>
[0]PETSC ERROR: TSStep() line 2625 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/interface/ts.c<br>
[0]PETSC ERROR: TSSolve() line 2741 in<br>
/home/mc/Downloads/petsc/src/<u></u>ts/interface/ts.c<br>
[0]PETSC ERROR: main() line 83 in<br>
/home/mc/PhD/opencl_tests/<u></u>petsc_opencl/petsc_opencl.cpp<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>
with errorcode -473550369.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
------------------------------<u></u>------------------------------<u></u>--------------<br>
<br>
The error occured at VecViennaCLGetArrayRead/Write in my<br>
ComputeResidualViennaCL function. Note that with the petsc debug mode,<br>
the code crashes with the above error even if I don't catch the error<br>
codes. Might be because of some VecViennaCLGetArray calls inside petsc.<br>
<br>
Cheers,<br>
Mani<br>
<br>
<br>
On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a><br></div></div><div class="im">
<mailto:<a href="mailto:rupp@mcs.anl.gov" target="_blank">rupp@mcs.anl.gov</a>>> wrote:<br>
<br>
Hi Mani,<br>
<br>
please check the return value of *all* function calls from PETSc, e.g.<br>
ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);<br>
instead of just<br>
DMCreateGlobalVector(da, &soln);<br>
Most likely one of the routines threw an error, but your code just<br>
kept going, producing wrong results.<br>
<br>
Best regards,<br>
Karli<br>
<br>
<br>
<br>
On 01/25/2014 05:35 AM, Mani Chandra wrote:<br>
<br>
Hi Everyone,<br>
<br>
I'm trying to use TS with ViennaCL vecs/mats and residual<br>
evaluation on<br>
device and have encountered some problems. I have attached a<br>
small test<br>
code that illustrates the issue.<br>
<br>
The code simply advects a blob diagonally using TS. I have<br>
written the<br>
residual evaluation function using 1) the usual Petsc vectors<br>
(VecGetArray) and 2) using ViennaCL vectors<br></div>
(VecViennaCLGetArrayRead/__<u></u>Write).<div class="im"><br>
<br>
Run the code using the following:<br>
./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type<br>
theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000<br>
-ts_monitor_draw_solution<br>
<br>
Case 1) No ViennaCL anywhere. I simply use the usual Petsc<br>
vectors and<br>
set the residual evaluation function as ComputeResidual (line<br>
no. 55).<br>
This case works and the blob is indeed advected as can be seen. (I<br>
haven't bothered with the boundaries. The simulation just stops<br>
before<br>
the blob hits the boundaries).<br>
<br>
Case 2) We again use the ComputeResidual but now enable ViennaCL<br>
vecs<br>
and mats (line nos. 48, 49). This case does NOT work. The SNES<br>
monitor<br>
shows convergence but the solution makes no sense.<br>
<br>
Case 3) We now use ComputeResidualViennaCL (line no. 56). This<br>
does NOT<br>
work either with or without enabling the ViennaCL vecs (line<br>
nos. 48, 49).<br>
<br>
Cheers,<br>
Mani<br>
<br>
<br>
<br>
</div></blockquote>
<br>
</blockquote></div><br></div>