On Tue, Feb 7, 2012 at 11:20 AM, recrusader <span dir="ltr"><<a href="mailto:recrusader@gmail.com">recrusader@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Whether is it possible to find an efficient mechanism to do the conversion between std::complex and cusp::complex when the conversion is necessary.<br></blockquote><div><br></div><div>That does not matter. This is a compile error. We are not going to change this right now, and it seems like you are not going</div>
<div>make the necessary changes, so I would say that complex numbers are not supported with our GPU code right now. The</div><div>change would involve using cusp::complex for PetscScalar, and I am not sure how much work that would entail.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks.<br>Yujie<br><br><div class="gmail_quote">On Tue, Feb 7, 2012 at 11:17 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Tue, Feb 7, 2012 at 11:15 AM, recrusader <span dir="ltr"><<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The following is the reply from Thrust developers,<br>"<span>The issue here is that the member functions of std::complex are
not decorated with __host__ __device__ and therefore the compiler
complains when asked to instantiate such functions in other __host__
__device__ code. If you switch std::complex to cusp::complex (which
*is* decorated with __host__ __device__) then the problem should
disappear.</span>"<br></blockquote><div><br></div></div><div>That makes sense, and my reply is the same. PETSc does not support that complex type. You can try to typedef</div><div>PetscScalar to that type, but I have no idea what problems it might cause.</div>
<div><br></div><div> Matt</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks.<br><br><div class="gmail_quote">On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On Tue, Feb 7, 2012 at 10:19 AM, recrusader <span dir="ltr"><<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>></span> wrote:<br></div><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear Matt,<br><br>I got the help from Thrust developers. <br>The problem is <br>in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar, cusp::device_memory>. Generaly, PetscScalar is std::complex.<br>However, the CUSPARRAY array cannot be decorated at GPU side. We need cusp::array1d<cusp::complex, cusp::device_memory>.<br>
Is there any simple method to process this in PETSc?<br></blockquote><div><br></div></div><div>This reply does not make any sense to me. What do you mean by the word "decorated"? If you mean that they only support the</div>
<div>complex type cusp::complex, then I advise you to configure for real numbers. We do not support cusp::complex as our complex</div><div>type.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div>
<div><div><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thank you very much.<br><br>Best,<br>Yujie<br><br><div class="gmail_quote">On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Sun, Jan 29, 2012 at 1:05 PM, recrusader <span dir="ltr"><<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thank you very much, Matt,<br>
<br>
You mean the headers of the simple codes, I further simply the codes as</blockquote><div><br></div></div><div>This is a question for the CUSP mailing list.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div>
<div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">"<br>
#include <cusp/blas.h><br>
#include <cusp/array1d.h><br>
<br>
int main(void)<br>
{<br>
cusp::array1d<std::complex<double>, cusp::host_memory> *x;<br>
<br>
x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);<br>
<br>
std::complex<double> alpha(1,2.0);<br>
cusp::blas::scal(*x,alpha);<br>
<br>
return 0;<br>
}"<br>
<br>
I got the same compilation results "<br>
login1$ nvcc <a href="http://gputest.cu" target="_blank">gputest.cu</a> -o gputest<br>
/opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):<br>
warning: calling a __host__ function from a __host__ __device__<br>
function is not allowed<br>
detected during:<br>
instantiation of "void<br>
cusp::blas::detail::SCAL<T>::operator()(T2 &) [with<br>
T=std::complex<double>, T2=std::complex<double>]"<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37): here<br>
instantiation of "InputIterator<br>
thrust::detail::host::for_each(InputIterator, InputIterator,<br>
UnaryFunction) [with<br>
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):<br>
here<br>
instantiation of "InputIterator<br>
thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
UnaryFunction, thrust::host_space_tag) [with<br>
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51): here<br>
instantiation of "InputIterator<br>
thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
[with InputIterator=thrust::detail::normal_iterator<std::complex<double><br>
*>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
/opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67): here<br>
instantiation of "void thrust::for_each(InputIterator,<br>
InputIterator, UnaryFunction) [with<br>
InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
(367): here<br>
instantiation of "void<br>
cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
[with ForwardIterator=thrust::detail::normal_iterator<std::complex<double><br>
*>, ScalarType=std::complex<double>]"<br>
(748): here<br>
instantiation of "void cusp::blas::scal(Array &,<br>
ScalarType) [with Array=cusp::array1d<std::complex<double>,<br>
cusp::host_memory>, ScalarType=std::complex<double>]"<br>
<a href="http://gputest.cu" target="_blank">gputest.cu</a>(25): here<br>
<br>
"<br>
<br>
Thanks a lot.<br>
<br>
Best,<br>
Yujie<br>
<br>
On 1/29/12, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Sun, Jan 29, 2012 at 12:53 PM, recrusader <<a href="mailto:recrusader@gmail.com" target="_blank">recrusader@gmail.com</a>> wrote:<br>
><br>
>> Dear PETSc developers,<br>
>><br>
>> With your help, I can successfully PETSc-deve with enabling GPU and<br>
>> complex number.<br>
>> However, when I compiled the codes, I met some errors. I also tried to<br>
>> use simple codes to realize the same function. However, the errors<br>
>> disappear. One example is as follows:<br>
>><br>
>> for the function "VecScale_SeqCUSP"<br>
>> "#undef __FUNCT__<br>
>> #define __FUNCT__ "VecScale_SeqCUSP"<br>
>> PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha)<br>
>> {<br>
>> CUSPARRAY *xarray;<br>
>> PetscErrorCode ierr;<br>
>><br>
>> PetscFunctionBegin;<br>
>> if (alpha == 0.0) {<br>
>> ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr);<br>
>> } else if (alpha != 1.0) {<br>
>> ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr);<br>
>> try {<br>
>> cusp::blas::scal(*xarray,alpha);<br>
>> } catch(char* ex) {<br>
>> SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex);<br>
>> }<br>
>> ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr);<br>
>> }<br>
>> ierr = WaitForGPU();CHKERRCUSP(ierr);<br>
>> ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr);<br>
>> PetscFunctionReturn(0);<br>
>> } "<br>
>><br>
>> When I compiled PETSc-dev, I met the following errors:<br>
>> " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134): warning:<br>
>> calling a __host__ function from a __host__ __device__ function is not<br>
>> allowed<br>
>> detected during:<br>
>> instantiation of "void<br>
>> cusp::blas::detail::SCAL<T>::operator()(T2 &) [with<br>
>> T=std::complex<double>, T2=PetscScalar]"<br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):<br>
>> here<br>
>> instantiation of "void<br>
>> thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,<br>
>> Size, UnaryFunction>::operator()() [with<br>
>><br>
>> RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> Size=long, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>><br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
>> here<br>
>> instantiation of "void<br>
>><br>
>> thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)<br>
>> [with<br>
>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> long, cusp::blas::detail::SCAL<std::complex<double>>>]"<br>
>><br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):<br>
>> here<br>
>> instantiation of "size_t<br>
>><br>
>> thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,<br>
>> launch_by_value>::block_size_with_maximal_occupancy(size_t) [with<br>
>><br>
>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> long, cusp::blas::detail::SCAL<std::complex<double>>>,<br>
>> launch_by_value=true]"<br>
>><br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):<br>
>> here<br>
>> instantiation of "thrust::pair<size_t, size_t><br>
>><br>
>> thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)<br>
>> [with<br>
>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"<br>
>><br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):<br>
>> here<br>
>> [ 6 instantiation contexts not shown ]<br>
>> instantiation of "InputIterator<br>
>> thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
>> UnaryFunction, thrust::device_space_tag) [with<br>
>><br>
>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here<br>
>> instantiation of "InputIterator<br>
>> thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
>> [with<br>
>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here<br>
>> instantiation of "void thrust::for_each(InputIterator,<br>
>> InputIterator, UnaryFunction) [with<br>
>><br>
>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> (367): here<br>
>> instantiation of "void<br>
>> cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
>> [with<br>
>> ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,<br>
>> ScalarType=std::complex<double>]"<br>
>> (748): here<br>
>> instantiation of "void cusp::blas::scal(Array &,<br>
>> ScalarType) [with Array=cusp::array1d<PetscScalar,<br>
>> cusp::device_memory>, ScalarType=std::complex<double>]"<br>
>> <a href="http://veccusp.cu" target="_blank">veccusp.cu</a>(1185): here<br>
>><br>
>><br>
>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
>> error: a value of type "int" cannot be assigned to an entity of type<br>
>> "_ZNSt7complexIdE9_ComplexTE"<br>
>><br>
>> "<br>
>> However, I further realize simiar codes as<br>
>> "<br>
>> #include <thrust/version.h><br>
>> #include <cusp/version.h><br>
>> #include <iostream><br>
>> #include <cusp/blas.h><br>
>> #include <cusp/array1d.h><br>
>> #include <complex><br>
>><br>
>> int main(void)<br>
>> {<br>
>> cusp::array1d<std::complex<double>, cusp::host_memory> *x;<br>
>><br>
>> x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);<br>
>><br>
>> std::complex<double> alpha(1,2.0);<br>
>> cusp::blas::scal(*x,alpha);<br>
>><br>
>> return 0;<br>
>> }<br>
>> "<br>
>><br>
>> When I complied it using "nvcc <a href="http://gputest.cu" target="_blank">gputest.cu</a> -o gputest", I only meet<br>
>> warning information as follows:<br>
>> "<br>
>> /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):<br>
>> warning: calling a __host__ function from a __host__ __device__<br>
>> function is not allowed<br>
>> detected during:<br>
>> instantiation of "void<br>
>> cusp::blas::detail::SCAL<T>::operator()(T2 &) [with<br>
>> T=std::complex<double>, T2=std::complex<double>]"<br>
>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):<br>
>> here<br>
>> instantiation of "InputIterator<br>
>> thrust::detail::host::for_each(InputIterator, InputIterator,<br>
>> UnaryFunction) [with<br>
>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>><br>
>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):<br>
>> here<br>
>> instantiation of "InputIterator<br>
>> thrust::detail::dispatch::for_each(InputIterator, InputIterator,<br>
>> UnaryFunction, thrust::host_space_tag) [with<br>
>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):<br>
>> here<br>
>> instantiation of "InputIterator<br>
>> thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)<br>
>> [with InputIterator=thrust::detail::normal_iterator<std::complex<double><br>
>> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):<br>
>> here<br>
>> instantiation of "void thrust::for_each(InputIterator,<br>
>> InputIterator, UnaryFunction) [with<br>
>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,<br>
>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"<br>
>> (367): here<br>
>> instantiation of "void<br>
>> cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)<br>
>> [with ForwardIterator=thrust::detail::normal_iterator<std::complex<double><br>
>> *>, ScalarType=std::complex<double>]"<br>
>> (748): here<br>
>> instantiation of "void cusp::blas::scal(Array &,<br>
>> ScalarType) [with Array=cusp::array1d<std::complex<double>,<br>
>> cusp::host_memory>, ScalarType=std::complex<double>]"<br>
>> <a href="http://gputest.cu" target="_blank">gputest.cu</a>(25): here<br>
>><br>
>> "<br>
>> There are not errors like<br>
>><br>
>> "/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):<br>
>> error: a value of type "int" cannot be assigned to an entity of type<br>
>> "_ZNSt7complexIdE9_ComplexTE" "<br>
>><br>
>> Furthermore, the warning information is also different between<br>
>> PETSc-dev and simple codes.<br>
>><br>
>> Could you give me some suggestion for this errors? Thank you very much.<br>
>><br>
><br>
> The headers are complicated to get right. The whole point of what we did is<br>
> to give a way to use GPU<br>
> simply through the existing PETSc linear algebra interface.<br>
><br>
> Matt<br>
><br>
><br>
>> Best,<br>
>> Yujie<br>
>><br>
><br>
><br>
<span><font color="#888888">><br>
> --<br>
> What most experimenters take for granted before they begin their<br>
> experiments is infinitely more interesting than any results to which their<br>
> experiments lead.<br>
> -- Norbert Wiener<br>
><br>
</font></span></blockquote></div></div></div><div><div><br><br clear="all"><span><font color="#888888"><span><font color="#888888"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></font></span></div></div></blockquote></div><span><font color="#888888"><br>
</font></span></blockquote></div></div></div><span><font color="#888888"><div><div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></div></div></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><div><div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</div></div></font></span></blockquote></div><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>