[petsc-dev] Possible bugs when using TS with ViennaCL (continued)

Tue Jan 28 01:07:54 CST 2014

Hi Karl,

I've been testing further, the code using TS with ViennaCL and there are a
couple of things I wanted to point out

1) When using the ComputeResidualViennaCL with either the normal Petsc
Vecs/Mats or Vec/MatViennaCL, and using the GPU, the nonlinear convergence
is very different from using an OpenCL CPU backend or just the regular
Petsc code.

a) Using NVIDIA OpenCL to run on the GPU to compute the residual and using
either normal Petsc Vec/Mat or ViennaCL Vec/Mat:

0 TS dt 10 time 0
    0 SNES Function norm 4.789374470711e-01
    1 SNES Function norm 5.491749197245e-02
    2 SNES Function norm 6.542412564158e-03
    3 SNES Function norm 7.800844032317e-04
    4 SNES Function norm 9.349243191537e-05
    5 SNES Function norm 1.120692741097e-05
1 TS dt 10 time 10

b) Using Intel OpenCL to run on the CPU to compute the residual and using
either normal Petsc Vec/Mat or ViennaCL Vec/Mat::

0 TS dt 10 time 0
    0 SNES Function norm 3.916582465172e-02
    1 SNES Function norm 4.990998832000e-07

c) Using ComputeResidual (which runs on the CPU) with the normal Petsc
Vec/Mat

0 TS dt 10 time 0
    0 SNES Function norm 3.916582465172e-02
    1 SNES Function norm 4.990998832000e-07
1 TS dt 10 time 10

You see that b) and c) match perfectly but a) is quite different. Why could
this be?

I also tried with the option -pc_type none cause I thought the GPU
preconditioners don't exist yet and I still get the above.

You can try with the attached code. I bumped up DOF to 8 and N1 and N2 to
128 each.
Run command:
./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 100 -ts_type theta
-ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000 -pc_type none

2) When I try using either ComputeResidual or ComputeResidualViennaCL with
the ViennaCL Vec/Mats, the GPU run crashes at a late time because of a
routine in ViennaCL.

ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
ViennaCL: Smaller work sizes could not solve the problem.
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
CL_MEM_OBJECT_ALLOCATION_FAILURE

I have attached the full crash log. The crash occurs late into the run, in
this case at the 80th time step. I thought all memory allocation occurs at
the beginning of the run, so I don't quite understand why its failing. Note
that the code works if I use ComputeResidualViennaCL with the normal Petsc
Vec/Mats.

Cheers,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/8f9f760d/attachment.html>
-------------- next part --------------
ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
ViennaCL: Smaller work sizes could not solve the problem. 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE 
 ViennaCL could not allocate memory on the device. Most likely the device simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at viennacl-support at lists.sourceforge.net and supply at least the following information:
 * Operating System
 * Which OpenCL implementation (AMD, NVIDIA, etc.)
 * ViennaCL version
Many thanks in advance!!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2368-g025cfdf  GIT Date: 2014-01-26 11:19:25 +0100
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc_opencl on a arch-linux2-c-debug named Deathstar by mc Tue Jan 28 00:24:44 2014
[0]PETSC ERROR: Libraries linked from /home/mc/Downloads/petsc_float_optimized/lib
[0]PETSC ERROR: Configure run at Mon Jan 27 17:37:55 2014
[0]PETSC ERROR: Configure options --prefix=/home/mc/Downloads/petsc_float_optimized/ --with-precision=single --with-debugging=0 COPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -qarch=native" --with-clean=1 --with-cuda=1 --with-opencl=1 --with-cuda-dir=/opt/cuda --download-txpetscgpu=yes --with-viennacl --download-viennacl make PETSC_DIR=/home/mc/Downloads/petsc PETSC_ARCH=arch-linux2-c-debug all
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatMult_SeqAIJViennaCL() line 196 in /home/mc/Downloads/petsc/src/mat/impls/aij/seq/seqviennacl/aijviennacl.cxx
[0]PETSC ERROR: MatMult() line 2242 in /home/mc/Downloads/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: PCApplyBAorAB() line 677 in /home/mc/Downloads/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApplyBAorAB() line 257 in /home/mc/Downloads/petsc/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPGMRESCycle() line 155 in /home/mc/Downloads/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 235 in /home/mc/Downloads/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 432 in /home/mc/Downloads/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 233 in /home/mc/Downloads/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3812 in /home/mc/Downloads/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_Theta() line 183 in /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: TSStep() line 2625 in /home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2741 in /home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: main() line 90 in /home/mc/PhD/opencl_tests/petsc_opencl_fixed/petsc_opencl.cpp
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 76.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl_fixed.tar.gz
Type: application/x-gzip
Size: 4978 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/8f9f760d/attachment.gz>