On Tue, Feb 7, 2012 at 10:19 AM, recrusader <span dir="ltr">&lt;<a href="mailto:recrusader@gmail.com">recrusader@gmail.com</a>&gt;</span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear Matt,<br><br>I got the help from Thrust developers. <br>The problem is <br>in PETSc, we define CUSPARRAY using cusp::array1d&lt;PetscScalar, cusp::device_memory&gt;. Generaly, PetscScalar is std::complex.<br>However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d&lt;cusp::complex, cusp::device_memory&gt;.<br>

Is there any simple method to process this in PETSc?<br></blockquote><div><br></div><div>This reply does not make any sense to me. What do you mean by the word &quot;decorated&quot;? If you mean that they only support the</div>
<div>complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex</div><div>type.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div><div>
 </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thank you very much.<br><br>Best,<br>Yujie<br><br><div class="gmail_quote">On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <span dir="ltr">&lt;<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Sun, Jan 29, 2012 at 1:05 PM, recrusader <span dir="ltr">&lt;<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>&gt;</span> wrote:<br>

</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thank you very much, Matt,<br>
<br>
You mean the headers of the simple codes, I further simply the codes as</blockquote><div><br></div></div><div>This is a question for the CUSP mailing list.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div>

<div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">&quot;<br>
#include &lt;cusp/blas.h&gt;<br>
#include &lt;cusp/array1d.h&gt;<br>
<br>
int main(void)<br>
{<br>
    cusp::array1d&lt;std::complex&lt;double&gt;, cusp::host_memory&gt; *x;<br>
<br>
    x=new cusp::array1d&lt;std::complex&lt;double&gt;, cusp::host_memory&gt;(2,0.0);<br>
<br>
    std::complex&lt;double&gt; alpha(1,2.0);<br>
    cusp::blas::scal(*x,alpha);<br>
<br>
    return 0;<br>
}&quot;<br>
<br>
I got the same compilation results &quot;<br>
login1$ nvcc <a href="http://gputest.cu" target="_blank">gputest.cu</a> -o gputest<br>
/opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):<br>
warning: calling a __host__ function from a __host__ __device__<br>
function is not allowed<br>
          detected during:<br>
            instantiation of &quot;void<br>
cusp::blas::detail::SCAL&lt;T&gt;::operator()(T2 &amp;) [with<br>
T=std::complex&lt;double&gt;, T2=std::complex&lt;double&gt;]&quot;<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here<br>
            instantiation of &quot;InputIterator<br>
thrust::detail::host::for_each(InputIterator, InputIterator,<br>
UnaryFunction) [with<br>
InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):<br>
here<br>
            instantiation of &quot;InputIterator<br>
thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
UnaryFunction, thrust::host_space_tag) [with<br>
InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here<br>
            instantiation of &quot;InputIterator<br>
thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
[with InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt;<br>
*&gt;, UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here<br>
            instantiation of &quot;void thrust::for_each(InputIterator,<br>
InputIterator, UnaryFunction) [with<br>
InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
(367): here<br>
            instantiation of &quot;void<br>
cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
[with ForwardIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt;<br>
*&gt;, ScalarType=std::complex&lt;double&gt;]&quot;<br>
(748): here<br>
            instantiation of &quot;void cusp::blas::scal(Array &amp;,<br>
ScalarType) [with Array=cusp::array1d&lt;std::complex&lt;double&gt;,<br>
cusp::host_memory&gt;, ScalarType=std::complex&lt;double&gt;]&quot;<br>
<a href="http://gputest.cu" target="_blank">gputest.cu</a>(25): here<br>
<br>
&quot;<br>
<br>
Thanks a lot.<br>
<br>
Best,<br>
Yujie<br>
<br>
On 1/29/12, Matthew Knepley &lt;<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>&gt; wrote:<br>
&gt; On Sun, Jan 29, 2012 at 12:53 PM, recrusader &lt;<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>&gt; wrote:<br>
&gt;<br>
&gt;&gt; Dear PETSc developers,<br>
&gt;&gt;<br>
&gt;&gt; With your help, I can successfully PETSc-deve with enabling GPU and<br>
&gt;&gt; complex number.<br>
&gt;&gt; However, when I compiled the codes, I met some errors. I also tried to<br>
&gt;&gt; use simple codes to realize the same function. However, the errors<br>
&gt;&gt; disappear. One example is as follows:<br>
&gt;&gt;<br>
&gt;&gt; for the function &quot;VecScale_SeqCUSP&quot;<br>
&gt;&gt; &quot;#undef __FUNCT__<br>
&gt;&gt; #define __FUNCT__ &quot;VecScale_SeqCUSP&quot;<br>
&gt;&gt; PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha)<br>
&gt;&gt; {<br>
&gt;&gt;  CUSPARRAY      *xarray;<br>
&gt;&gt;  PetscErrorCode ierr;<br>
&gt;&gt;<br>
&gt;&gt;  PetscFunctionBegin;<br>
&gt;&gt;  if (alpha == 0.0) {<br>
&gt;&gt;    ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr);<br>
&gt;&gt;  } else if (alpha != 1.0) {<br>
&gt;&gt;    ierr = VecCUSPGetArrayReadWrite(xin,&amp;xarray);CHKERRQ(ierr);<br>
&gt;&gt;    try {<br>
&gt;&gt;      cusp::blas::scal(*xarray,alpha);<br>
&gt;&gt;    } catch(char* ex) {<br>
&gt;&gt;      SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,&quot;CUSP error: %s&quot;, ex);<br>
&gt;&gt;    }<br>
&gt;&gt;    ierr = VecCUSPRestoreArrayReadWrite(xin,&amp;xarray);CHKERRQ(ierr);<br>
&gt;&gt;  }<br>
&gt;&gt;  ierr = WaitForGPU();CHKERRCUSP(ierr);<br>
&gt;&gt;  ierr = PetscLogFlops(xin-&gt;map-&gt;n);CHKERRQ(ierr);<br>
&gt;&gt;  PetscFunctionReturn(0);<br>
&gt;&gt; } &quot;<br>
&gt;&gt;<br>
&gt;&gt; When I compiled PETSc-dev, I met the following errors:<br>
&gt;&gt; &quot; /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning:<br>
&gt;&gt; calling a __host__ function from a __host__ __device__ function is not<br>
&gt;&gt; allowed<br>
&gt;&gt;          detected during:<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt; cusp::blas::detail::SCAL&lt;T&gt;::operator()(T2 &amp;) [with<br>
&gt;&gt; T=std::complex&lt;double&gt;, T2=PetscScalar]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt; thrust::detail::device::cuda::for_each_n_closure&lt;RandomAccessIterator,<br>
&gt;&gt; Size, UnaryFunction&gt;::operator()() [with<br>
&gt;&gt;<br>
&gt;&gt; RandomAccessIterator=thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; Size=long, UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt;<br>
&gt;&gt; thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)<br>
&gt;&gt; [with<br>
&gt;&gt; NullaryFunction=thrust::detail::device::cuda::for_each_n_closure&lt;thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; long, cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;&gt;]&quot;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;size_t<br>
&gt;&gt;<br>
&gt;&gt; thrust::detail::device::cuda::detail::closure_launcher_base&lt;NullaryFunction,<br>
&gt;&gt; launch_by_value&gt;::block_size_with_maximal_occupancy(size_t) [with<br>
&gt;&gt;<br>
&gt;&gt; NullaryFunction=thrust::detail::device::cuda::for_each_n_closure&lt;thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; long, cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;&gt;,<br>
&gt;&gt; launch_by_value=true]&quot;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;thrust::pair&lt;size_t, size_t&gt;<br>
&gt;&gt;<br>
&gt;&gt; thrust::detail::device::cuda::detail::closure_launcher&lt;NullaryFunction&gt;::configuration_with_maximal_occupancy(Size)<br>
&gt;&gt; [with<br>
&gt;&gt; NullaryFunction=thrust::detail::device::cuda::for_each_n_closure&lt;thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; long, cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;&gt;, Size=long]&quot;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):<br>
&gt;&gt; here<br>
&gt;&gt;            [ 6 instantiation contexts not shown ]<br>
&gt;&gt;            instantiation of &quot;InputIterator<br>
&gt;&gt; thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
&gt;&gt; UnaryFunction, thrust::device_space_tag) [with<br>
&gt;&gt;<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here<br>
&gt;&gt;            instantiation of &quot;InputIterator<br>
&gt;&gt; thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
&gt;&gt; [with<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here<br>
&gt;&gt;            instantiation of &quot;void thrust::for_each(InputIterator,<br>
&gt;&gt; InputIterator, UnaryFunction) [with<br>
&gt;&gt;<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; (367): here<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt; cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
&gt;&gt; [with<br>
&gt;&gt; ForwardIterator=thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;PetscScalar&gt;&gt;,<br>
&gt;&gt; ScalarType=std::complex&lt;double&gt;]&quot;<br>
&gt;&gt; (748): here<br>
&gt;&gt;            instantiation of &quot;void cusp::blas::scal(Array &amp;,<br>
&gt;&gt; ScalarType) [with Array=cusp::array1d&lt;PetscScalar,<br>
&gt;&gt; cusp::device_memory&gt;, ScalarType=std::complex&lt;double&gt;]&quot;<br>
&gt;&gt; <a href="http://veccusp.cu" target="_blank">veccusp.cu</a>(1185): here<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
&gt;&gt; error: a value of type &quot;int&quot; cannot be assigned to an entity of type<br>
&gt;&gt; &quot;_ZNSt7complexIdE9_ComplexTE&quot;<br>
&gt;&gt;<br>
&gt;&gt; &quot;<br>
&gt;&gt; However, I further realize simiar codes as<br>
&gt;&gt; &quot;<br>
&gt;&gt; #include &lt;thrust/version.h&gt;<br>
&gt;&gt; #include &lt;cusp/version.h&gt;<br>
&gt;&gt; #include &lt;iostream&gt;<br>
&gt;&gt; #include &lt;cusp/blas.h&gt;<br>
&gt;&gt; #include &lt;cusp/array1d.h&gt;<br>
&gt;&gt; #include &lt;complex&gt;<br>
&gt;&gt;<br>
&gt;&gt; int main(void)<br>
&gt;&gt; {<br>
&gt;&gt;    cusp::array1d&lt;std::complex&lt;double&gt;, cusp::host_memory&gt; *x;<br>
&gt;&gt;<br>
&gt;&gt;    x=new cusp::array1d&lt;std::complex&lt;double&gt;, cusp::host_memory&gt;(2,0.0);<br>
&gt;&gt;<br>
&gt;&gt;    std::complex&lt;double&gt; alpha(1,2.0);<br>
&gt;&gt;    cusp::blas::scal(*x,alpha);<br>
&gt;&gt;<br>
&gt;&gt;    return 0;<br>
&gt;&gt; }<br>
&gt;&gt; &quot;<br>
&gt;&gt;<br>
&gt;&gt; When I complied it using  &quot;nvcc <a href="http://gputest.cu" target="_blank">gputest.cu</a> -o gputest&quot;, I only meet<br>
&gt;&gt; warning information as follows:<br>
&gt;&gt; &quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):<br>
&gt;&gt; warning: calling a __host__ function from a __host__ __device__<br>
&gt;&gt; function is not allowed<br>
&gt;&gt;          detected during:<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt; cusp::blas::detail::SCAL&lt;T&gt;::operator()(T2 &amp;) [with<br>
&gt;&gt; T=std::complex&lt;double&gt;, T2=std::complex&lt;double&gt;]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;InputIterator<br>
&gt;&gt; thrust::detail::host::for_each(InputIterator, InputIterator,<br>
&gt;&gt; UnaryFunction) [with<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;InputIterator<br>
&gt;&gt; thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
&gt;&gt; UnaryFunction, thrust::host_space_tag) [with<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;InputIterator<br>
&gt;&gt; thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
&gt;&gt; [with InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt;<br>
&gt;&gt; *&gt;, UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):<br>
&gt;&gt; here<br>
&gt;&gt;            instantiation of &quot;void thrust::for_each(InputIterator,<br>
&gt;&gt; InputIterator, UnaryFunction) [with<br>
&gt;&gt; InputIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt; *&gt;,<br>
&gt;&gt; UnaryFunction=cusp::blas::detail::SCAL&lt;std::complex&lt;double&gt;&gt;]&quot;<br>
&gt;&gt; (367): here<br>
&gt;&gt;            instantiation of &quot;void<br>
&gt;&gt; cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
&gt;&gt; [with ForwardIterator=thrust::detail::normal_iterator&lt;std::complex&lt;double&gt;<br>
&gt;&gt; *&gt;, ScalarType=std::complex&lt;double&gt;]&quot;<br>
&gt;&gt; (748): here<br>
&gt;&gt;            instantiation of &quot;void cusp::blas::scal(Array &amp;,<br>
&gt;&gt; ScalarType) [with Array=cusp::array1d&lt;std::complex&lt;double&gt;,<br>
&gt;&gt; cusp::host_memory&gt;, ScalarType=std::complex&lt;double&gt;]&quot;<br>
&gt;&gt; <a href="http://gputest.cu" target="_blank">gputest.cu</a>(25): here<br>
&gt;&gt;<br>
&gt;&gt; &quot;<br>
&gt;&gt; There are not errors like<br>
&gt;&gt;<br>
&gt;&gt; &quot;/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
&gt;&gt; error: a value of type &quot;int&quot; cannot be assigned to an entity of type<br>
&gt;&gt; &quot;_ZNSt7complexIdE9_ComplexTE&quot; &quot;<br>
&gt;&gt;<br>
&gt;&gt; Furthermore, the warning information is also different between<br>
&gt;&gt; PETSc-dev and simple codes.<br>
&gt;&gt;<br>
&gt;&gt; Could you give me some suggestion for this errors? Thank you very much.<br>
&gt;&gt;<br>
&gt;<br>
&gt; The headers are complicated to get right. The whole point of what we did is<br>
&gt; to give a way to use GPU<br>
&gt; simply through the existing PETSc linear algebra interface.<br>
&gt;<br>
&gt;    Matt<br>
&gt;<br>
&gt;<br>
&gt;&gt; Best,<br>
&gt;&gt; Yujie<br>
&gt;&gt;<br>
&gt;<br>
&gt;<br>
<span><font color="#888888">&gt;<br>
&gt; --<br>
&gt; What most experimenters take for granted before they begin their<br>
&gt; experiments is infinitely more interesting than any results to which their<br>
&gt; experiments lead.<br>
&gt; -- Norbert Wiener<br>
&gt;<br>
</font></span></blockquote></div></div></div><div><div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>


-- Norbert Wiener<br>
</font></span></div></div></blockquote></div><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>