<div dir="ltr">Well, it has some problems but it is good that we have it so we can check that box off. And if anyone ever picks AMGx up again PETSc might be the best place to do that.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 13, 2019 at 2:28 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks, Mark,<div><br></div><div>it looks really cool!</div><div><br></div><div>Fande,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 13, 2019 at 11:29 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">There is a MR winding its way in. I think it is ready to go. There is confusion with branches but it is something like barry/12-01-19-pc-feature-amgx.<div><br></div><div>It is a PC, but it is so fragile. I could only get AMGx to work with classical AMG (preferred) by using its Krylov solver also. So the test sets -ksp_preonly. </div><div><br></div><div>There is an example in snes/ex13.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 13, 2019 at 12:09 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I was wondering how we use AMGx? It is a PC? Do we have a native interface to this package? Any example?<div><br></div><div>Fande,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 3, 2019 at 9:10 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Ok, since we are using a fork of the repository if it is causing grief we can remove it but otherwise best just to leave it since it put in by the AMGX developers.<br>
<br>
<br>
<br>
> On Dec 3, 2019, at 10:00 PM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> <br>
> Its from the attached configure.log [from a prior e-mail on this<br>
> thread] - and this flag is not passed in from petsc configure to amgx<br>
> cmake - so it must be somehow set internally in this package.<br>
> <br>
> Satish<br>
> <br>
> On Wed, 4 Dec 2019, Smith, Barry F. wrote:<br>
> <br>
>> <br>
>>> Also - its best to avoid -Werror in externalpackage builds..<br>
>> <br>
>> Hmm, my branch uses the CMake package to do the install (it has no custom code) so is this a bug in package.py that it doesn't strip out the -Werror when passing stuff to cmake?<br>
>> <br>
>> There is not enough information your email for me to determine where this message you printed below came from. Did it come from my branch?<br>
>> <br>
>> Barry<br>
>> <br>
>> <br>
>>> On Dec 3, 2019, at 9:29 PM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
>>> <br>
>>>>>>> <br>
>>> autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.15.2-xit2o3iepxvqbyku77lwcugufilztu7t/bin/cmake -E remove /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/core/CMakeFiles/amgx_core.dir/src/classical/interpolators/./<a href="http://amgx_core_generated_common.cu" target="_blank">amgx_core_generated_common.cu</a>.o<br>
>>> /sw/summit/cuda/10.1.168/bin/nvcc -M -D__CUDACC__ /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/<a href="http://plugin_config.cu" rel="noreferrer" target="_blank">plugin_config.cu</a> -o /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/base/CMakeFiles/amgx_base.dir/__/amgx_base_generated_plugin_config.cu.o.NVCC-depend -m64 -Xcompiler ,\"-fstack-protector\",\"-g\",\"-O2\",\"-fPIC\",\"-Wno-terminate\",\"-static-libgcc\",\"-fopenmp\",\"-DRAPIDJSON_DEFINED\",\"-DAMGX_WITH_MPI\",\"-fstack-protector\",\"-g\",\"-O2\",\"-fPIC\" -ccbin mpicxx -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_52,code=\"sm_52,compute_52\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch=compute_70,code=\"sm_70,compute_70\" -O3 -DNDEBUG -std=c++11 --Werror cross-execution-space-call -DNVCC -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/../../thrust -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git..amgx/base/include -I/sw/summit/cuda/10.1.168/include -I/autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/external/rapidjson/include<br>
>>> -- Removing /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/petsc-build/base/CMakeFiles/amgx_base.dir/src/./amgx_base_generated_device_properties.cu.o<br>
>>> <<<<<<<br>
>>> <br>
>>> Also - its best to avoid -Werror in externalpackage builds..<br>
>>> <br>
>>> Satish<br>
>>> <br>
>>> On Tue, 3 Dec 2019, Mark Adams wrote:<br>
>>> <br>
>>>> Barry,<br>
>>>> <br>
>>>> First, there is a fix that we need in the AMGx repo (appended).<br>
>>>> <br>
>>>> I've never had problems like this not being able to build. I will try to<br>
>>>> build my AMGx manually to check.<br>
>>>> <br>
>>>> 21:31 master *= ~/AMGX$ git diff<br>
>>>> diff --git a/base/include/amgx_c_wrappers.inl<br>
>>>> b/base/include/amgx_c_wrappers.inl<br>
>>>> index 42496f0..ed9f0e1 100644<br>
>>>> --- a/base/include/amgx_c_wrappers.inl<br>
>>>> +++ b/base/include/amgx_c_wrappers.inl<br>
>>>> @@ -643,7 +643,7 @@ inline AMGX_Mode get_mode_from(const Envelope &envl)<br>
>>>> {<br>
>>>> //throws...<br>
>>>> //<br>
>>>> - FatalError("Mode not found.\n", AMGX_ERR_BAD_MODE);<br>
>>>> + // FatalError("Mode not found.\n", AMGX_ERR_BAD_MODE);<br>
>>>> }<br>
>>>> <br>
>>>> AMGX_Mode mode = static_cast<AMGX_Mode>(itFound->second);<br>
>>>> @@ -1125,4 +1125,4 @@ inline bool remove_managed_matrix(AMGX_matrix_handle<br>
>>>> envl)<br>
>>>> } //namespace unnamed<br>
>>>> <br>
>>>> On Tue, Dec 3, 2019 at 8:50 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
>>>> <br>
>>>>> <br>
>>>>> The first error is<br>
>>>>> <br>
>>>>> nvcc error : 'cicc' died due to signal 9 (Kill signal)<br>
>>>>> nvcc error : 'cicc' died due to signal 9 (Kill signal)<br>
>>>>> <br>
>>>>> later<br>
>>>>> <br>
>>>>> <br>
>>>>> /autofs/nccs-svm1_home1/adams/petsc/arch-summit-opt64-gnu-cuda/externalpackages/git.amgx/base/src/<br>
>>>>> <a href="http://amgx_c_common.cu" rel="noreferrer" target="_blank">amgx_c_common.cu</a>(77): catastrophic error: error while writing generated<br>
>>>>> C++ file: Cannot allocate memory<br>
>>>>> <br>
>>>>> 1 catastrophic error detected in the compilation of<br>
>>>>> "/tmp/tmpxft_000038d3_00000000-4_amgx_c_common.cpp4.ii".<br>
>>>>> <br>
>>>>> Whatever machine you are compiling on seems to be overloaded. What does<br>
>>>>> top show? Can you log into "different" compiler servers on Summit?<br>
>>>>> <br>
>>>>> I run the install on the utk system with no problems like this.<br>
>>>>> <br>
>>>>> PETSc configure is deciding<br>
>>>>> <br>
>>>>> TEST configureMakeNP from<br>
>>>>> config.packages.make(/autofs/nccs-svm1_home1/adams/petsc/config/BuildSystem/config/packages/make.py:168)<br>
>>>>> TESTING: configureMakeNP from<br>
>>>>> config.packages.make(config/BuildSystem/config/packages/make.py:168)<br>
>>>>> check no of cores on the build machine [perhaps to do make '-j ncores']<br>
>>>>> module multiprocessing found 128 cores: using make_np = 59<br>
>>>>> Defined make macro "MAKE_NP" to "59"<br>
>>>>> Defined make macro "MAKE_TEST_NP" to "49"<br>
>>>>> Defined make macro "MAKE_LOAD" to "166.4"<br>
>>>>> Defined make macro "NPMAX" to "128"<br>
>>>>> <br>
>>>>> then somehow this gets passed down to the AMGX cmake. Perhaps you could<br>
>>>>> try less greedy numbers.<br>
>>>>> <br>
>>>>> Executing: /usr/bin/gmake -j59 -l166.4<br>
>>>>> <br>
>>>>> Perhaps try<br>
>>>>> <br>
>>>>> --with-make-np=20<br>
>>>>> <br>
>>>>> --with-make-load=20<br>
>>>>> <br>
>>>>> Barry<br>
>>>>> <br>
>>>>> <br>
>>>>> <br>
>>>>>> On Dec 3, 2019, at 4:17 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> <br>
>>>>>> That might have been from an old .nsf file. Here is a new one. I'm sure<br>
>>>>> I have the disk space.<br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 4:51 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> Humm, clean up and seem to have different error:<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 3:45 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
>>>>> wrote:<br>
>>>>>> Could you have run out of disk space?<br>
>>>>>> <br>
>>>>>> /usr/bin/ranlib: libamgx_base.a: Input/output error<br>
>>>>>> gmake[2]: *** [base/libamgx_base.a] Error 1<br>
>>>>>> gmake[2]: *** Deleting file `base/libamgx_base.a'<br>
>>>>>> gmake[1]: *** [base/CMakeFiles/amgx_base.dir/all] Error 2<br>
>>>>>> gmake: *** [all] Error 2<br>
>>>>>> <br>
>>>>>> Matt<br>
>>>>>> <br>
>>>>>> On Tue, Dec 3, 2019 at 1:44 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> <br>
>>>>>> --<br>
>>>>>> What most experimenters take for granted before they begin their<br>
>>>>> experiments is infinitely more interesting than any results to which their<br>
>>>>> experiments lead.<br>
>>>>>> -- Norbert Wiener<br>
>>>>>> <br>
>>>>>> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>>>>>> <configure.log><br>
>>>>> <br>
>>>>> <br>
>>>> <br>
>> <br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>