<p>UNRESTRICTED / ILLIMITÉE</p><html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The scenario that I considered involves a structured grid of size 32 x 32 x 32 (using Pflotran here). When I used the bcgs solver and the preconditioner jacobi, there were no issues, but when the preconditioner sacusp was used I received the following error:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>terminate called after throwing an instance of 'thrust::system::system_error'<o:p></o:p></p><p class=MsoNormal> what(): function_attributes(): after cudaFuncGetAttributes: an illegal memory access was encountered.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am attaching the screen shots of my debugger (idb) that will provide some detail regarding the source of this error. My Petsc configuration is <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>--download-mpich=yes --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-shared-libraries=0 --with-debugging=1 --with-valgrind=1 --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-fblaslapack=yes --with-c2html=0 --with-cuda=1 --with-cusp=1 --with-thrust=1 --with-64-bit-indices=1<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have to use 64 bit indices, if I avoid it, I receive out of memory issues. Machines used are Intel Xeon E5-2620 and Tesla K20c card having the compute capability of 3.5. When I reduced the grid size to 16 x 16 x 16 I no more receive the above error with sacusp.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Can you point me what could be the problem here.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Regards,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Rajeev Das.<o:p></o:p></p></div></body></html>