Ran on behalf of Satish on Cookie with a single Tesla 2070 Fermi.<br><br><br>Configure Options: --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --with-mpi-dir=/disks/soft/mpich2-1.3.1-gcc --download-f-blas-lapack=yes --with-cuda-dir=/soft/cuda-4.0/cuda --with-thrust-dir=/soft/cuda-4.0/cuda/include --with-cusp-dir=/soft/cuda-4.0/cuda/include -with-debugging=0 --with-cudac=nvcc --with-precision=double --with-clanguage=c --with-cuda-arch=sm_20 PETSC_ARCH=structgrid_cuda<br>
<br><br>./ex19 -da_vec_type seqcusp -da_mat_type seqaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -mat_no_inode -preload off -cusp_synchronize -cuda_set_device 0 -log_summary ex19.cudaCookie2070Fermi.log<br>
<br>lid velocity = 0.0001, prandtl # = 1, grashof # = 1<br>Number of SNES iterations = 2<br><br><br><br><br><br><br><div class="gmail_quote">On Mon, Sep 19, 2011 at 11:10 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><br>
Satish,<br>
<br>
Thanks<br>
<br>
Ok, those are all what we expect: so what the hey is wrong with Shiyuan machine? Is there another machine you can try on?<br>
<br>
<br>
VecDot 2 1.0 1.2088e-04 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1324<br>
VecMDot 2024 1.0 2.1392e+00 1.0 2.54e+09 1.0 0.0e+00 0.0e+00 0.0e+00 35 29 0 0 0 49 29 0 0 0 1189<br>
VecNorm 2096 1.0 1.1928e-01 1.0 1.68e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 3 2 0 0 0 1406<br>
VecScale 2092 1.0 5.2948e-02 1.0 8.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580<br>
VecCopy 2072 1.0 6.9294e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0<br>
VecSet 70 1.0 1.7152e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecAXPY 108 1.0 1.3336e-02 1.0 8.64e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 648<br>
VecWAXPY 68 1.0 2.0800e-03 1.0 2.72e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1308<br>
VecMAXPY 2092 1.0 2.9396e-01 1.0 2.71e+09 1.0 0.0e+00 0.0e+00 0.0e+00 5 31 0 0 0 7 31 0 0 0 9205<br>
VecScatterBegin 5 1.0 1.7390e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecReduceArith 2 1.0 9.2983e-04 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 172<br>
VecReduceComm 1 1.0 2.1458e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecCUSPCopyTo 49 1.0 9.9115e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
VecCUSPCopyFrom 44 1.0 1.4175e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
SNESSolve 1 1.0 4.3493e+00 1.0 8.87e+09 1.0 0.0e+00 0.0e+00 3.4e+04 71100 0 0100 100100 0 0100 2040<br>
SNESLineSearch 2 1.0 8.4569e-03 1.0 5.49e+06 1.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 650<br>
SNESFunctionEval 3 1.0 6.6361e-03 1.0 2.52e+06 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 380<br>
SNESJacobianEval 2 1.0 5.0695e-01 1.0 3.85e+07 1.0 0.0e+00 0.0e+00 4.3e+01 8 0 0 0 0 12 0 0 0 0 76<br>
KSPGMRESOrthog 2024 1.0 2.4282e+00 1.0 5.09e+09 1.0 0.0e+00 0.0e+00 3.1e+04 40 57 0 0 92 56 57 0 0 93 2095<br>
KSPSetup 2 1.0 2.9206e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 0 0<br>
KSPSolve 2 1.0 3.8301e+00 1.0 8.83e+09 1.0 0.0e+00 0.0e+00 3.4e+04 63 99 0 0100 88 99 0 0100 2304<br>
PCSetUp 2 1.0 2.1458e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
PCApply 2024 1.0 6.3726e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0<br>
MatMult 2092 1.0 1.1330e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 19 37 0 0 0 26 37 0 0 0 2931<br>
<br>
Barry<br>
<div><div></div><div class="h5"><br>
On Sep 19, 2011, at 10:44 AM, Satish Balay wrote:<br>
<br>
> Attached is the output from the run on breadboard. It has 2 "nVidia<br>
> Corporation GT200 [Tesla C1060]" cards.<br>
><br>
> Satish<br>
><br>
> --------<br>
><br>
> balay@bb30:~/petsc-dev/src/snes/examples/tutorials>./ex19 -da_vec_type seqcusp -da_mat_type seqaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -mat_no_inode -preload off -cusp_synchronize -cuda_set_device 0 -log_summary ex19.cuda.log<br>
> lid velocity = 0.0001, prandtl # = 1, grashof # = 1<br>
> Number of SNES iterations = 2<br>
> balay@bb30:~/petsc-dev/src/snes/examples/tutorials><br>
><br>
> On Sun, 18 Sep 2011, Barry Smith wrote:<br>
><br>
>><br>
>><br>
>> Ok, the copy up and down are not a problem.<br>
>><br>
>> Except for VecMAXPY() the vector operations are terrible (like they are not using the GPU, but they must be?) The MatMult() must be GPU because it is pretty good 2779???<br>
>><br>
>> Does someone else have access to a similar system and can they run the exact same test to see what numbers they get? Satish, could you on breadboard? Maybe on Magellion :-)<br>
>><br>
>><br>
>> Barry<br>
>><br>
>><br>
>><br>
>> VecDot 2 1.0 1.7049e-03 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 94<br>
>> VecMDot 2024 1.0 8.6273e+00 1.0 2.54e+09 1.0 0.0e+00 0.0e+00 0.0e+00 50 29 0 0 0 66 29 0 0 0 295<br>
>> VecNorm 2096 1.0 1.5544e+00 1.0 1.68e+08 1.0 0.0e+00 0.0e+00 0.0e+00 9 2 0 0 0 12 2 0 0 0 108<br>
>> VecScale 2092 1.0 3.7774e-01 1.0 8.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 3 1 0 0 0 222<br>
>> VecCopy 2072 1.0 3.8258e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 3 0 0 0 0 0<br>
>> VecSet 70 1.0 1.3119e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> VecAXPY 108 1.0 4.7407e-02 1.0 8.64e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 182<br>
>> VecWAXPY 68 1.0 1.2545e-02 1.0 2.72e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 217<br>
>> VecMAXPY 2092 1.0 6.4464e-01 1.0 2.71e+09 1.0 0.0e+00 0.0e+00 0.0e+00 4 31 0 0 0 5 31 0 0 0 4198<br>
>> VecScatterBegin 5 1.0 1.5609e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> VecReduceArith 2 1.0 3.8650e-03 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41<br>
>> VecReduceComm 1 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> VecCUSPCopyTo 49 1.0 3.0950e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> VecCUSPCopyFrom 44 1.0 2.0876e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> SNESSolve 1 1.0 1.3044e+01 1.0 8.87e+09 1.0 0.0e+00 0.0e+00 0.0e+00 75100 0 0 0 100100 0 0 0 680<br>
>> SNESLineSearch 2 1.0 1.1921e-02 1.0 5.49e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 461<br>
>> SNESFunctionEval 3 1.0 2.7192e-03 1.0 2.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 927<br>
>> SNESJacobianEval 2 1.0 2.0424e-01 1.0 3.85e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 188<br>
>> KSPGMRESOrthog 2024 1.0 9.2522e+00 1.0 5.09e+09 1.0 0.0e+00 0.0e+00 0.0e+00 53 57 0 0 0 71 57 0 0 0 550<br>
>> KSPSetup 2 1.0 5.1975e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> KSPSolve 2 1.0 1.2819e+01 1.0 8.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 74 99 0 0 0 98 99 0 0 0 689<br>
>> PCSetUp 2 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>> PCApply 2024 1.0 3.8054e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 3 0 0 0 0 0<br>
>> MatMult 2092 1.0 1.1950e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 7 37 0 0 0 9 37 0 0 0 2779<br>
>><br>
>> On Sep 18, 2011, at 10:29 AM, Shiyuan wrote:<br>
>><br>
>>><br>
>>><br>
>>> On Sat, Sep 17, 2011 at 10:48 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
>>><br>
>>> Run the first one with -da_vec_type seqcusp and -da_mat_type seqaijcusp<br>
>>><br>
>>>> VecScatterBegin 2097 1.0 1.0270e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 7 0 0 0 0 0<br>
>>>> VecCUSPCopyTo 2140 1.0 2.4991e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0<br>
>>>> VecCUSPCopyFrom 2135 1.0 1.0437e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 7 0 0 0 0 0<br>
>>><br>
>>> Why is it doing all these vector copy ups and downs? It is run on one process it shouldn't be doing more than a handful total.<br>
>>><br>
>>> Barry<br>
>>><br>
>>> ./ex19 -da_vec_type seqcusp -da_mat_type seqaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cusp_synchronize -cuda_set_device 0 | tee ex19p2.txt<br>
>>><br>
>>> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --<br>
>>> Avg %Total Avg %Total counts %Total Avg %Total counts %Total<br>
>>> 0: Main Stage: 4.2393e+00 24.4% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%<br>
>>> 1: SetUp: 4.9079e-02 0.3% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%<br>
>>> 2: Solve: 1.3071e+01 75.3% 8.8712e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%<br>
>>><br>
>>> ------------------------------------------------------------------------------------------------------------------------<br>
>>><br>
>>> VecScatterBegin 5 1.0 1.5609e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>>> VecReduceArith 2 1.0 3.8650e-03 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41<br>
>>> VecReduceComm 1 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>>> VecCUSPCopyTo 49 1.0 3.0950e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>>> VecCUSPCopyFrom 44 1.0 2.0876e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
>>><br>
>>> The complete log is attached. Thanks.<br>
>>> <ex19p2.txt><br>
>><br>
>><br>
</div></div>> <ex19.cuda.log><br>
<br>
</blockquote></div><br>