<div dir="ltr">There is a MR winding its way in. I think it is ready to go. There is confusion with branches but it is something like barry/12-01-19-pc-feature-amgx.<div><br></div><div>It is a PC, but it is so fragile. I could only get AMGx to work with classical AMG (preferred) by using its Krylov solver also. So the test sets -ksp_preonly. </div><div><br></div><div>There is an example in snes/ex13.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 13, 2019 at 12:09 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I was wondering how we use AMGx? It is a PC? Do we have a native interface to this package? Any example?<div><br></div><div>Fande,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 3, 2019 at 9:10 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Ok, since we are using a fork of the repository if it is causing grief we can remove it but otherwise best just to leave it since it put in by the AMGX developers.<br>
<br>
<br>
<br>
> On Dec 3, 2019, at 10:00 PM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> <br>
> Its from the attached configure.log [from a prior e-mail on this<br>
> thread] - and this flag is not passed in from petsc configure to amgx<br>
> cmake - so it must be somehow set internally in this package.<br>
> <br>
> Satish<br>
> <br>
> On Wed, 4 Dec 2019, Smith, Barry F. wrote:<br>
> <br>
>> <br>
>>> Also - its best to avoid -Werror in externalpackage builds..<br>
>> <br>
>> Hmm, my branch uses the CMake package to do the install (it has no custom code) so is this a bug in package.py that it doesn't strip out the -Werror when passing stuff to cmake?<br>
>> <br>
>> There is not enough information your email for me to determine where this message you printed below came from. Did it come from my branch?<br>
>> <br>
>> Barry<br>
>> <br>
>> <br>
>>> On Dec 3, 2019, at 9:29 PM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
>>> <br>
>>>>>>> <br>
>>> autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.15.2-xit2o3iepxvqbyku77lwcugufilztu7t/bin/cmake -E remove /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/core/CMakeFiles/amgx_core.dir/src/classical/interpolators/./<a href="http://amgx_core_generated_common.cu" target="_blank">amgx_core_generated_common.cu</a>.o<br>
>>> /sw/summit/cuda/10.1.168/bin/nvcc -M -D__CUDACC__ /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/<a href="http://plugin_config.cu" rel="noreferrer" target="_blank">plugin_config.cu</a> -o /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/base/CMakeFiles/amgx_base.dir/__/amgx_base_generated_plugin_config.cu.o.NVCC-depend -m64 -Xcompiler ,\"-fstack-protector\",\"-g\",\"-O2\",\"-fPIC\",\"-Wno-terminate\",\"-static-libgcc\",\"-fopenmp\",\"-DRAPIDJSON_DEFINED\",\"-DAMGX_WITH_MPI\",\"-fstack-protector\",\"-g\",\"-O2\",\"-fPIC\" -ccbin mpicxx -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -O3 -DNDEBUG -std=c++11 --Werror cross-execution-space-call -DNVCC -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/../../thrust -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git..amgx/base/include -I/sw/summit/cuda/10.1.168/include -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/external/rapidjson/include<br>
>>> -- Removing /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/base/CMakeFiles/amgx_base.dir/src/./amgx_base_generated_device_properties.cu.o<br>
>>> <<<<<<<br>
>>> <br>
>>> Also - its best to avoid -Werror in externalpackage builds..<br>
>>> <br>
>>> Satish<br>
>>> <br>
>>> On Tue, 3 Dec 2019, Mark Adams wrote:<br>
>>> <br>
>>>> Barry,<br>
>>>> <br>
>>>> First, there is a fix that we need in the AMGx repo (appended).<br>
>>>> <br>
>>>> I've never had problems like this not being able to build. I will try to<br>
>>>> build my AMGx manually to check.<br>
>>>> <br>
>>>> 21:31 master *= ~/AMGX$ git diff<br>
>>>> diff --git a/base/include/amgx_c_wrappers.inl<br>
>>>> b/base/include/amgx_c_wrappers.inl<br>
>>>> index 42496f0..ed9f0e1 100644<br>
>>>> --- a/base/include/amgx_c_wrappers.inl<br>
>>>> +++ b/base/include/amgx_c_wrappers.inl<br>
>>>> @@ -643,7 +643,7 @@ inline AMGX_Mode get_mode_from(const Envelope &envl)<br>
>>>> {<br>
>>>> //throws...<br>
>>>> //<br>
>>>> - FatalError("Mode not found.\n", AMGX_ERR_BAD_MODE);<br>
>>>> + // FatalError("Mode not found.\n", AMGX_ERR_BAD_MODE);<br>
>>>> }<br>
>>>> <br>
>>>> AMGX_Mode mode = static_cast<AMGX_Mode>(itFound->second);<br>
>>>> @@ -1125,4 +1125,4 @@ inline bool remove_managed_matrix(AMGX_matrix_handle<br>
>>>> envl)<br>
>>>> } //namespace unnamed<br>
>>>> <br>
>>>> On Tue, Dec 3, 2019 at 8:50 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
>>>> <br>
>>>>> <br>
>>>>> The first error is<br>
>>>>> <br>
>>>>> nvcc error : 'cicc' died due to signal 9 (Kill signal)<br>
>>>>> nvcc error : 'cicc' died due to signal 9 (Kill signal)<br>
>>>>> <br>
>>>>> later<br>
>>>>> <br>
>>>>> <br>
>>>>> /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/base/src/<br>
>>>>> <a href="http://amgx_c_common.cu" rel="noreferrer" target="_blank">amgx_c_common.cu</a>(77): catastrophic error: error while writing generated<br>
>>>>> C++ file: Cannot allocate memory<br>
>>>>> <br>
>>>>> 1 catastrophic error detected in the compilation of<br>
>>>>> "/tmp/tmpxft_000038d3_00000000-4_amgx_c_common.cpp4.ii".<br>
>>>>> <br>
>>>>> Whatever machine you are compiling on seems to be overloaded. What does<br>
>>>>> top show? Can you log into "different" compiler servers on Summit?<br>
>>>>> <br>
>>>>> I run the install on the utk system with no problems like this.<br>
>>>>> <br>
>>>>> PETSc configure is deciding<br>
>>>>> <br>
>>>>> TEST configureMakeNP from<br>
>>>>> config.packages.make(/autofs/nccs-svm1_home1/adams/petsc/config/BuildSystem/config/packages/make.py:168)<br>
>>>>> TESTING: configureMakeNP from<br>
>>>>> config.packages.make(config/BuildSystem/config/packages/make.py:168)<br>
>>>>> check no of cores on the build machine [perhaps to do make '-j ncores']<br>
>>>>> module multiprocessing found 128 cores: using make_np = 59<br>
>>>>> Defined make macro "MAKE_NP" to "59"<br>
>>>>> Defined make macro "MAKE_TEST_NP" to "49"<br>
>>>>> Defined make macro "MAKE_LOAD" to "166.4"<br>
>>>>> Defined make macro "NPMAX" to "128"<br>
>>>>> <br>
>>>>> then somehow this gets passed down to the AMGX cmake. Perhaps you could<br>
>>>>> try less greedy numbers.<br>
>>>>> <br>
>>>>> Executing: /usr/bin/gmake -j59 -l166.4<br>
>>>>> <br>
>>>>> Perhaps try<br>
>>>>> <br>
>>>>> --with-make-np=20<br>
>>>>> <br>
>>>>> --with-make-load=20<br>
>>>>> <br>
>>>>> Barry<br>
>>>>> <br>
>>>>> <br>
>>>>> <br>
>>>>>> On Dec 3, 2019, at 4:17 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> <br>
>>>>>> That might have been from an old .nsf file. Here is a new one. I'm sure<br>
>>>>> I have the disk space.<br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 4:51 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> Humm, clean up and seem to have different error:<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 3:45 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
>>>>> wrote:<br>
>>>>>> Could you have run out of disk space?<br>
>>>>>> <br>
>>>>>> /usr/bin/ranlib: libamgx_base.a: Input/output error<br>
>>>>>> gmake[2]: *** [base/libamgx_base.a] Error 1<br>
>>>>>> gmake[2]: *** Deleting file `base/libamgx_base.a'<br>
>>>>>> gmake[1]: *** [base/CMakeFiles/amgx_base.dir/all] Error 2<br>
>>>>>> gmake: *** [all] Error 2<br>
>>>>>> <br>
>>>>>> Matt<br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 1:44 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> <br>
>>>>>> --<br>
>>>>>> What most experimenters take for granted before they begin their<br>
>>>>> experiments is infinitely more interesting than any results to which their<br>
>>>>> experiments lead.<br>
>>>>>> -- Norbert Wiener<br>
>>>>>> <br>
>>>>>> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>>>>>> <configure.log><br>
>>>>> <br>
>>>>> <br>
>>>> <br>
>> <br>
<br>
</blockquote></div>
</blockquote></div>