[petsc-users] one compilation error in PETSc-dev with enabling GPU and complex number
Matthew Knepley
knepley at gmail.com
Tue Feb 7 11:17:47 CST 2012
On Tue, Feb 7, 2012 at 11:15 AM, recrusader <recrusader at gmail.com> wrote:
> The following is the reply from Thrust developers,
> "The issue here is that the member functions of std::complex are not
> decorated with __host__ __device__ and therefore the compiler complains
> when asked to instantiate such functions in other __host__ __device__ code.
> If you switch std::complex to cusp::complex (which *is* decorated with
> __host__ __device__) then the problem should disappear."
>
That makes sense, and my reply is the same. PETSc does not support that
complex type. You can try to typedef
PetscScalar to that type, but I have no idea what problems it might cause.
Matt
> Thanks.
>
> On Tue, Feb 7, 2012 at 11:13 AM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Feb 7, 2012 at 10:19 AM, recrusader <recrusader at gmail.com> wrote:
>>
>>> Dear Matt,
>>>
>>> I got the help from Thrust developers.
>>> The problem is
>>> in PETSc, we define CUSPARRAY using cusp::array1d<PetscScalar,
>>> cusp::device_memory>. Generaly, PetscScalar is std::complex.
>>> However, the CUSPARRAY array cannot be decorated at GPU side. We need
>>> cusp::array1d<cusp::complex, cusp::device_memory>.
>>> Is there any simple method to process this in PETSc?
>>>
>>
>> This reply does not make any sense to me. What do you mean by the word
>> "decorated"? If you mean that they only support the
>> complex type cusp::complex, then I advise you to configure for real
>> numbers. We do not support cusp::complex as our complex
>> type.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Thank you very much.
>>>
>>> Best,
>>> Yujie
>>>
>>> On Sun, Jan 29, 2012 at 1:20 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Sun, Jan 29, 2012 at 1:05 PM, recrusader <recrusader at gmail.com>wrote:
>>>>
>>>>> Thank you very much, Matt,
>>>>>
>>>>> You mean the headers of the simple codes, I further simply the codes as
>>>>
>>>>
>>>> This is a question for the CUSP mailing list.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> "
>>>>> #include <cusp/blas.h>
>>>>> #include <cusp/array1d.h>
>>>>>
>>>>> int main(void)
>>>>> {
>>>>> cusp::array1d<std::complex<double>, cusp::host_memory> *x;
>>>>>
>>>>> x=new cusp::array1d<std::complex<double>, cusp::host_memory>(2,0.0);
>>>>>
>>>>> std::complex<double> alpha(1,2.0);
>>>>> cusp::blas::scal(*x,alpha);
>>>>>
>>>>> return 0;
>>>>> }"
>>>>>
>>>>> I got the same compilation results "
>>>>> login1$ nvcc gputest.cu -o gputest
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):
>>>>> warning: calling a __host__ function from a __host__ __device__
>>>>> function is not allowed
>>>>> detected during:
>>>>> instantiation of "void
>>>>> cusp::blas::detail::SCAL<T>::operator()(T2 &) [with
>>>>> T=std::complex<double>, T2=std::complex<double>]"
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
>>>>> here
>>>>> instantiation of "InputIterator
>>>>> thrust::detail::host::for_each(InputIterator, InputIterator,
>>>>> UnaryFunction) [with
>>>>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
>>>>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>>
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
>>>>> here
>>>>> instantiation of "InputIterator
>>>>> thrust::detail::dispatch::for_each(InputIterator, InputIterator,
>>>>> UnaryFunction, thrust::host_space_tag) [with
>>>>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
>>>>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
>>>>> here
>>>>> instantiation of "InputIterator
>>>>> thrust::detail::for_each(InputIterator, InputIterator, UnaryFunction)
>>>>> [with
>>>>> InputIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
>>>>> here
>>>>> instantiation of "void thrust::for_each(InputIterator,
>>>>> InputIterator, UnaryFunction) [with
>>>>> InputIterator=thrust::detail::normal_iterator<std::complex<double> *>,
>>>>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> (367): here
>>>>> instantiation of "void
>>>>> cusp::blas::detail::scal(ForwardIterator, ForwardIterator, ScalarType)
>>>>> [with
>>>>> ForwardIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> *>, ScalarType=std::complex<double>]"
>>>>> (748): here
>>>>> instantiation of "void cusp::blas::scal(Array &,
>>>>> ScalarType) [with Array=cusp::array1d<std::complex<double>,
>>>>> cusp::host_memory>, ScalarType=std::complex<double>]"
>>>>> gputest.cu(25): here
>>>>>
>>>>> "
>>>>>
>>>>> Thanks a lot.
>>>>>
>>>>> Best,
>>>>> Yujie
>>>>>
>>>>> On 1/29/12, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> > On Sun, Jan 29, 2012 at 12:53 PM, recrusader <recrusader at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> >> Dear PETSc developers,
>>>>> >>
>>>>> >> With your help, I can successfully PETSc-deve with enabling GPU and
>>>>> >> complex number.
>>>>> >> However, when I compiled the codes, I met some errors. I also tried
>>>>> to
>>>>> >> use simple codes to realize the same function. However, the errors
>>>>> >> disappear. One example is as follows:
>>>>> >>
>>>>> >> for the function "VecScale_SeqCUSP"
>>>>> >> "#undef __FUNCT__
>>>>> >> #define __FUNCT__ "VecScale_SeqCUSP"
>>>>> >> PetscErrorCode VecScale_SeqCUSP(Vec xin, PetscScalar alpha)
>>>>> >> {
>>>>> >> CUSPARRAY *xarray;
>>>>> >> PetscErrorCode ierr;
>>>>> >>
>>>>> >> PetscFunctionBegin;
>>>>> >> if (alpha == 0.0) {
>>>>> >> ierr = VecSet_SeqCUSP(xin,alpha);CHKERRQ(ierr);
>>>>> >> } else if (alpha != 1.0) {
>>>>> >> ierr = VecCUSPGetArrayReadWrite(xin,&xarray);CHKERRQ(ierr);
>>>>> >> try {
>>>>> >> cusp::blas::scal(*xarray,alpha);
>>>>> >> } catch(char* ex) {
>>>>> >> SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUSP error: %s", ex);
>>>>> >> }
>>>>> >> ierr = VecCUSPRestoreArrayReadWrite(xin,&xarray);CHKERRQ(ierr);
>>>>> >> }
>>>>> >> ierr = WaitForGPU();CHKERRCUSP(ierr);
>>>>> >> ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr);
>>>>> >> PetscFunctionReturn(0);
>>>>> >> } "
>>>>> >>
>>>>> >> When I compiled PETSc-dev, I met the following errors:
>>>>> >> " /opt/apps/cuda/4.0/cuda/include/cusp/detail/blas.inl(134):
>>>>> warning:
>>>>> >> calling a __host__ function from a __host__ __device__ function is
>>>>> not
>>>>> >> allowed
>>>>> >> detected during:
>>>>> >> instantiation of "void
>>>>> >> cusp::blas::detail::SCAL<T>::operator()(T2 &) [with
>>>>> >> T=std::complex<double>, T2=PetscScalar]"
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/for_each.inl(72):
>>>>> >> here
>>>>> >> instantiation of "void
>>>>> >>
>>>>> thrust::detail::device::cuda::for_each_n_closure<RandomAccessIterator,
>>>>> >> Size, UnaryFunction>::operator()() [with
>>>>> >>
>>>>> >>
>>>>> RandomAccessIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> Size=long,
>>>>> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
>>>>> >> here
>>>>> >> instantiation of "void
>>>>> >>
>>>>> >>
>>>>> thrust::detail::device::cuda::detail::launch_closure_by_value(NullaryFunction)
>>>>> >> [with
>>>>> >>
>>>>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> long, cusp::blas::detail::SCAL<std::complex<double>>>]"
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(71):
>>>>> >> here
>>>>> >> instantiation of "size_t
>>>>> >>
>>>>> >>
>>>>> thrust::detail::device::cuda::detail::closure_launcher_base<NullaryFunction,
>>>>> >> launch_by_value>::block_size_with_maximal_occupancy(size_t) [with
>>>>> >>
>>>>> >>
>>>>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> long, cusp::blas::detail::SCAL<std::complex<double>>>,
>>>>> >> launch_by_value=true]"
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(136):
>>>>> >> here
>>>>> >> instantiation of "thrust::pair<size_t, size_t>
>>>>> >>
>>>>> >>
>>>>> thrust::detail::device::cuda::detail::closure_launcher<NullaryFunction>::configuration_with_maximal_occupancy(Size)
>>>>> >> [with
>>>>> >>
>>>>> NullaryFunction=thrust::detail::device::cuda::for_each_n_closure<thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> long, cusp::blas::detail::SCAL<std::complex<double>>>, Size=long]"
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(145):
>>>>> >> here
>>>>> >> [ 6 instantiation contexts not shown ]
>>>>> >> instantiation of "InputIterator
>>>>> >> thrust::detail::dispatch::for_each(InputIterator, InputIterator,
>>>>> >> UnaryFunction, thrust::device_space_tag) [with
>>>>> >>
>>>>> >>
>>>>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >> /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(51): here
>>>>> >> instantiation of "InputIterator
>>>>> >> thrust::detail::for_each(InputIterator, InputIterator,
>>>>> UnaryFunction)
>>>>> >> [with
>>>>> >>
>>>>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >> /opt/apps/cuda/4.0/cuda/include/thrust/detail/for_each.inl(67): here
>>>>> >> instantiation of "void thrust::for_each(InputIterator,
>>>>> >> InputIterator, UnaryFunction) [with
>>>>> >>
>>>>> >>
>>>>> InputIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >> (367): here
>>>>> >> instantiation of "void
>>>>> >> cusp::blas::detail::scal(ForwardIterator, ForwardIterator,
>>>>> ScalarType)
>>>>> >> [with
>>>>> >>
>>>>> ForwardIterator=thrust::detail::normal_iterator<thrust::device_ptr<PetscScalar>>,
>>>>> >> ScalarType=std::complex<double>]"
>>>>> >> (748): here
>>>>> >> instantiation of "void cusp::blas::scal(Array &,
>>>>> >> ScalarType) [with Array=cusp::array1d<PetscScalar,
>>>>> >> cusp::device_memory>, ScalarType=std::complex<double>]"
>>>>> >> veccusp.cu(1185): here
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
>>>>> >> error: a value of type "int" cannot be assigned to an entity of type
>>>>> >> "_ZNSt7complexIdE9_ComplexTE"
>>>>> >>
>>>>> >> "
>>>>> >> However, I further realize simiar codes as
>>>>> >> "
>>>>> >> #include <thrust/version.h>
>>>>> >> #include <cusp/version.h>
>>>>> >> #include <iostream>
>>>>> >> #include <cusp/blas.h>
>>>>> >> #include <cusp/array1d.h>
>>>>> >> #include <complex>
>>>>> >>
>>>>> >> int main(void)
>>>>> >> {
>>>>> >> cusp::array1d<std::complex<double>, cusp::host_memory> *x;
>>>>> >>
>>>>> >> x=new cusp::array1d<std::complex<double>,
>>>>> cusp::host_memory>(2,0.0);
>>>>> >>
>>>>> >> std::complex<double> alpha(1,2.0);
>>>>> >> cusp::blas::scal(*x,alpha);
>>>>> >>
>>>>> >> return 0;
>>>>> >> }
>>>>> >> "
>>>>> >>
>>>>> >> When I complied it using "nvcc gputest.cu -o gputest", I only meet
>>>>> >> warning information as follows:
>>>>> >> "
>>>>> >> /opt/apps/cuda/4.0/cuda/bin/../include/cusp/detail/blas.inl(134):
>>>>> >> warning: calling a __host__ function from a __host__ __device__
>>>>> >> function is not allowed
>>>>> >> detected during:
>>>>> >> instantiation of "void
>>>>> >> cusp::blas::detail::SCAL<T>::operator()(T2 &) [with
>>>>> >> T=std::complex<double>, T2=std::complex<double>]"
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/host/for_each.inl(37):
>>>>> >> here
>>>>> >> instantiation of "InputIterator
>>>>> >> thrust::detail::host::for_each(InputIterator, InputIterator,
>>>>> >> UnaryFunction) [with
>>>>> >> InputIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> *>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >>
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/dispatch/for_each.h(46):
>>>>> >> here
>>>>> >> instantiation of "InputIterator
>>>>> >> thrust::detail::dispatch::for_each(InputIterator, InputIterator,
>>>>> >> UnaryFunction, thrust::host_space_tag) [with
>>>>> >> InputIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> *>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(51):
>>>>> >> here
>>>>> >> instantiation of "InputIterator
>>>>> >> thrust::detail::for_each(InputIterator, InputIterator,
>>>>> UnaryFunction)
>>>>> >> [with
>>>>> InputIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> >> *>, UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >>
>>>>> /opt/apps/cuda/4.0/cuda/bin/../include/thrust/detail/for_each.inl(67):
>>>>> >> here
>>>>> >> instantiation of "void thrust::for_each(InputIterator,
>>>>> >> InputIterator, UnaryFunction) [with
>>>>> >> InputIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> *>,
>>>>> >> UnaryFunction=cusp::blas::detail::SCAL<std::complex<double>>]"
>>>>> >> (367): here
>>>>> >> instantiation of "void
>>>>> >> cusp::blas::detail::scal(ForwardIterator, ForwardIterator,
>>>>> ScalarType)
>>>>> >> [with
>>>>> ForwardIterator=thrust::detail::normal_iterator<std::complex<double>
>>>>> >> *>, ScalarType=std::complex<double>]"
>>>>> >> (748): here
>>>>> >> instantiation of "void cusp::blas::scal(Array &,
>>>>> >> ScalarType) [with Array=cusp::array1d<std::complex<double>,
>>>>> >> cusp::host_memory>, ScalarType=std::complex<double>]"
>>>>> >> gputest.cu(25): here
>>>>> >>
>>>>> >> "
>>>>> >> There are not errors like
>>>>> >>
>>>>> >>
>>>>> "/opt/apps/cuda/4.0/cuda/include/thrust/detail/device/cuda/detail/launch_closure.inl(51):
>>>>> >> error: a value of type "int" cannot be assigned to an entity of type
>>>>> >> "_ZNSt7complexIdE9_ComplexTE" "
>>>>> >>
>>>>> >> Furthermore, the warning information is also different between
>>>>> >> PETSc-dev and simple codes.
>>>>> >>
>>>>> >> Could you give me some suggestion for this errors? Thank you very
>>>>> much.
>>>>> >>
>>>>> >
>>>>> > The headers are complicated to get right. The whole point of what we
>>>>> did is
>>>>> > to give a way to use GPU
>>>>> > simply through the existing PETSc linear algebra interface.
>>>>> >
>>>>> > Matt
>>>>> >
>>>>> >
>>>>> >> Best,
>>>>> >> Yujie
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > What most experimenters take for granted before they begin their
>>>>> > experiments is infinitely more interesting than any results to which
>>>>> their
>>>>> > experiments lead.
>>>>> > -- Norbert Wiener
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120207/0e6e1630/attachment-0001.htm>
More information about the petsc-users
mailing list