<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello All,<div class=""><br class=""></div><div class="">I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21):</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><tt class=""><font size="2" class="">The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit).<br class="">The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching <br class="">the limit.<br class=""> <br class="">Can you please give the following source code workaround a try?<br class="">Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals <br class="">and all similar procedures.<br class=""> <br class="">For example, the test case has:<br class=""> module petscvecdef<br class=""> use petscvecdefdummy<br class="">...<br class=""> function vecnotequal(A,B)<br class=""> use petscvecdefdummy<br class=""> logical vecnotequal<br class=""> type(tVec), intent(in) :: A,B<br class=""> vecnotequal = (A%v .ne. B%v)<br class=""> end function<br class=""> function vecequals(A,B)<br class=""> use petscvecdefdummy<br class=""> logical vecequals<br class=""> type(tVec), intent(in) :: A,B<br class=""> vecequals = (A%v .eq. B%v)<br class=""> end function<br class="">...<br class="">end module<br class="">Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries.<br class=""> <br class=""> <br class="">Please let us know if one of the above workarounds resolve the issue.</font></tt></blockquote></div><div class=""><br class=""></div><div class="">The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible.</div><div class=""><blockquote type="cite" class=""><p class=""><font size="2" class="">IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them?</font><br class=""><br class=""><tt class=""><font size="2" class=""> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using </font></tt><tt class=""><b class=""><font size="2" class="">use <module>, only: <symbol> or import <symbol> </font></b></tt><tt class=""><font size="2" class="">. if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile.<br class=""> <br class="">For example<br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> use petscvecdef<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class=""> <br class="">imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either:<br class=""> <br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> use petscvecdef, </font></tt><tt class=""><b class=""><font size="2" class="">only: tVec</font></b></tt><tt class=""><font size="2" class=""><br class=""> implicit none<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class=""> <br class="">or if use petscvecdef is used in the outer scope, we can:<br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> </font></tt><tt class=""><b class=""><font size="2" class="">import tVec</font></b></tt><tt class=""><font size="2" class=""><br class=""> implicit none<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class="">(The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.)</font></tt></p></blockquote></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Is this compiler ~feature~ something that we intend to work around? Thoughts?</span></div><div class=""><br class=""></div><div class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">Begin forwarded message:</div><br class="Apple-interchange-newline"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">From: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">"Roy Musselman" <<a href="mailto:roymuss@us.ibm.com" class="">roymuss@us.ibm.com</a>><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Subject: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class=""><b class="">Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines</b><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Date: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">March 3, 2021 at 08:23:17 CST<br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">To: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Jacob Faibussowitsch <<a href="mailto:faibuss2@illinois.edu" class="">faibuss2@illinois.edu</a>><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Cc: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">"Gyllenhaal, John C." <<a href="mailto:gyllenhaal1@llnl.gov" class="">gyllenhaal1@llnl.gov</a>><br class=""></span></div><br class=""><div class=""><meta charset="UTF-8" class=""><p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" class="">Hi Jacob,<span class="Apple-converted-space"> </span></font><br class=""><font size="2" class="">I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library.</font><br class=""><br class=""><b class="">.../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition.</b><span class="Apple-converted-space"> </span><br class=""><br class=""><font size="2" class="">IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them?</font><br class=""><br class=""><tt class=""><font size="2" class=""> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using<span class="Apple-converted-space"> </span></font></tt><tt class=""><b class=""><font size="2" class="">use <module>, only: <symbol> or import <symbol><span class="Apple-converted-space"> </span></font></b></tt><tt class=""><font size="2" class="">. if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile.<br class=""> <br class="">For example<br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> use petscvecdef<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class=""> <br class="">imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either:<br class=""> <br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> use petscvecdef,<span class="Apple-converted-space"> </span></font></tt><tt class=""><b class=""><font size="2" class="">only: tVec</font></b></tt><tt class=""><font size="2" class=""><br class=""> implicit none<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class=""> <br class="">or if use petscvecdef is used in the outer scope, we can:<br class=""> Interface<br class=""> Subroutine VecRestoreArrayReadF90(v,array,ierr)<br class=""> </font></tt><tt class=""><b class=""><font size="2" class="">import tVec</font></b></tt><tt class=""><font size="2" class=""><br class=""> implicit none<br class=""> real(kind=selected_real_kind(10)), pointer :: array(:)<br class=""> integer(kind=selected_int_kind(5)) ierr<br class=""> type(tVec) v<br class=""> End Subroutine<br class=""> End Interface<br class="">(The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.)</font></tt><br class=""><br class=""><font size="2" class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab<br class="">email:<span class="Apple-converted-space"> </span><a href="mailto:roymuss@us.ibm.com" class="">roymuss@us.ibm.com</a><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><br class=""><br class=""><img border="0" alt="Inactive hide details for Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 a" class="" apple-inline="yes" id="6FFDBD03-1D34-4133-806F-9AA90C68CECA" src="cid:1__=8FBB0C1EDFDE1EE08f9e8a93df938690918c8FB@"><font size="2" color="#424282" class="">Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket</font><br class=""><br class=""><font size="2" color="#5F5F5F" class="">From:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">Roy Musselman/Rochester/Contr/IBM</font><br class=""><font size="2" color="#5F5F5F" class="">To:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">Jacob Faibussowitsch <<a href="mailto:faibuss2@illinois.edu" class="">faibuss2@illinois.edu</a>></font><br class=""><font size="2" color="#5F5F5F" class="">Cc:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">"Gyllenhaal, John C." <<a href="mailto:gyllenhaal1@llnl.gov" class="">gyllenhaal1@llnl.gov</a>></font><br class=""><font size="2" color="#5F5F5F" class="">Date:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">02/24/2021 07:08 PM</font><br class=""><font size="2" color="#5F5F5F" class="">Subject:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines</font><br class=""></p><hr width="100%" size="2" align="left" noshade="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(128, 145, 165);" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hi Jacob,<span class="Apple-converted-space"> </span></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></font><a href="https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues__;!!DZ3fjg!vDUpTg4q6jg1lQwt37jm9Uzc7MqGrEdrg0wpKgGq9P5JoR3jKrqncOAKyni2BEUYOxQ$" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><font size="2" class="">https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues</font></a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps?<span class="Apple-converted-space"> </span></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><tt style="caret-color: rgb(0, 0, 0); font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" class=""> The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit).<br class="">The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching<span class="Apple-converted-space"> </span><br class="">the limit.<br class=""> <br class="">Can you please give the following source code workaround a try?<br class="">Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals<span class="Apple-converted-space"> </span><br class="">and all similar procedures.<br class=""> <br class="">For example, the test case has:<br class=""> module petscvecdef<br class=""> use petscvecdefdummy<br class="">...<br class=""> function vecnotequal(A,B)<br class=""> use petscvecdefdummy<br class=""> logical vecnotequal<br class=""> type(tVec), intent(in) :: A,B<br class=""> vecnotequal = (A%v .ne. B%v)<br class=""> end function<br class=""> function vecequals(A,B)<br class=""> use petscvecdefdummy<br class=""> logical vecequals<br class=""> type(tVec), intent(in) :: A,B<br class=""> vecequals = (A%v .eq. B%v)<br class=""> end function<br class="">...<br class="">end module<br class="">Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries.<br class=""> <br class=""> <br class="">Please let us know if one of the above workarounds resolve the issue.</font></tt><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab<br class="">email:<span class="Apple-converted-space"> </span><a href="mailto:roymuss@us.ibm.com" class="">roymuss@us.ibm.com</a><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><img border="0" alt="Inactive hide details for Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may ha" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="" apple-inline="yes" id="1AFBC9D6-508E-4E6A-9CB9-C532DE0F9426" src="cid:1__=8FBB0C1EDFDE1EE08f9e8a93df938690918c8FB@"><font size="2" color="#424282" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">From:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Roy Musselman/Rochester/Contr/IBM</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">To:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Jacob Faibussowitsch <<a href="mailto:faibuss2@illinois.edu" class="">faibuss2@illinois.edu</a>></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Cc:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">"Gyllenhaal, John C." <<a href="mailto:gyllenhaal1@llnl.gov" class="">gyllenhaal1@llnl.gov</a>></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Date:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">02/21/2021 09:42 PM</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Subject:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><hr width="100%" size="2" align="left" noshade="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(128, 145, 165);" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hi Jacob,<span class="Apple-converted-space"> </span></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team.</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2.</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack.</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">[attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM]<span class="Apple-converted-space"> </span></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab<br class="">email:<span class="Apple-converted-space"> </span><a href="mailto:roymuss@us.ibm.com" class="">roymuss@us.ibm.com</a><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><img border="0" alt="Inactive hide details for Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are us" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="" apple-inline="yes" id="1D1B4693-55D9-4327-AF83-C3C023CFDFC9" src="cid:1__=8FBB0C1EDFDE1EE08f9e8a93df938690918c8FB@"><font size="2" color="#424282" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">From:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Jacob Faibussowitsch <<a href="mailto:faibuss2@illinois.edu" class="">faibuss2@illinois.edu</a>></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">To:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Roy Musselman <<a href="mailto:roymuss@us.ibm.com" class="">roymuss@us.ibm.com</a>></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Cc:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">"Gyllenhaal, John C." <<a href="mailto:gyllenhaal1@llnl.gov" class="">gyllenhaal1@llnl.gov</a>></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Date:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">02/21/2021 12:24 PM</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="2" color="#5F5F5F" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Subject:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">[EXTERNAL] Re: xlf90_r Internal Compiler Error</font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><hr width="100%" size="2" align="left" noshade="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; color: rgb(128, 145, 165);" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font size="1" color="#FFFFFF" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc<span class="Apple-converted-space"> </span></font><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Hi Roy,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version.</font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist.</font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">3.14.0 and 3.14.1</font><br class=""><font size="2" class=""><br class="">"../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module.<br class="">1501-511 Compilation failed for file petscvecmod.F90.</font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written.<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding).</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem.<span class="Apple-converted-space"> </span></font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this?<span class="Apple-converted-space"> </span></font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class=""><font size="2" class="">Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler?<span class="Apple-converted-space"> </span></font></ul></ul><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Best regards,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Jacob Faibussowitsch</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">(Jacob Fai - booss - oh - vitch)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM]<span class="Apple-converted-space"> </span></span><ul style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><ul class="">On Feb 20, 2021, at 22:05, Roy Musselman <<a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font color="#0000FF" class="">roymuss@us.ibm.com</font></u></a>> wrote:<br class=""><p class=""><font size="2" class="">Hi Jacob,<br class="">Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today.</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">> ml avail petsc<br class="">----------------------------------------------------- /usr/tcetmp/modulefiles/Core -----------------------------------------------------<br class="">petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D)<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><br class="">petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist.</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies.<span class="Apple-converted-space"> </span><br class="">The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem.<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">#!/bin/bash</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src"<span class="Apple-converted-space"> </span><br class="">OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include"<br class="">mkdir -p moddir</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">set -x<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class=""># Compile original source files including dependencies<br class="">if [ 0 = 1 ]; then<br class="">mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o<span class="Apple-converted-space"> </span><br class="">mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o<br class="">mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o<br class="">fi</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class=""># Use -E option to expand source into full source files<br class="">if [ 0 = 1 ]; then<br class="">mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90<br class="">mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90<br class="">mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90<br class="">fi</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class=""># Compile from full source files<br class="">if [ 1 = 1 ]; then<br class="">mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o<br class="">mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o<br class="">mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o<br class="">fi</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class=""><eof></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results:<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">3.14.0 and 3.14.1</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">"../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module.<br class="">1501-511 Compilation failed for file petscvecmod.F90.</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">3.14.2, 3.14.3, and 3.14.4</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">. . .<br class="">** matnullspaceequals === End of Compilation 8 ===<br class="">*** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 ***</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Examining the tail end of petscmatmod.F90</font><font size="2" class=""><br class=""><br class=""></font><font size="2" class=""><br class="">80 function matnullspaceequals(A,B)<br class="">81 use petscmatdefdummy<br class="">82 logical matnullspaceequals<br class="">83 type(tMatNullSpace), intent(in) :: A,B<br class="">84 matnullspaceequals = (A%v .eq. B%v)<br class="">85 end function<br class="">86<span class="Apple-converted-space"> </span><br class="">87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES)<br class="">88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal<br class="">89 !DEC$ ATTRIBUTES DLLEXPORT::matequals<br class="">90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal<br class="">91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals<br class="">92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal<br class="">93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals<br class="">94 #endif<br class="">95 module petscmat<br class="">96 use petscmatdef<br class="">97 use petscvec<br class="">98 #include <../src/mat/f90-mod/petscmat.h90><br class="">99 interface<br class="">100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90><br class="">101 end interface<br class="">102 end module<br class="">103<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4.<br class="">Just looking at the list of subroutines contained within each version, the following are the differences.<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Old subroutines available in 3.13.6 but removed from 4.14.4<br class="">subroutine MatFreeIntermediateDataStructures(a,z)</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">New subroutines available in 4.14.4 but not contained in 3.13.6<span class="Apple-converted-space"> </span><br class="">subroutine MatDenseReplaceArray(a,b,z)<br class="">subroutine MatIsShell(a,b,z)<br class="">subroutine MatRARtMultEqual(a,b,c,d,e,z)<br class="">subroutine MatScaLAPACKGetBlockSizes(a,b,c,z)<br class="">subroutine MatScaLAPACKSetBlockSizes(a,b,c,z)<br class="">subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z)<br class="">subroutine MatSeqAIJSetTotalPreallocation(a,b,z)<br class="">subroutine MatSetLayouts(a,b,c,z)</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters.<br class="">A little research found:<span class="Apple-converted-space"> </span></font><font size="2" class="">In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters.<span class="Apple-converted-space"> </span></font><font size="2" class="">I found the following subroutines with greater than 31 characters</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">subroutine MatCreateMPIMatConcatenateSeqMat<br class="">subroutine MatFactorFactorizeSchurComplement<br class="">subroutine MatMPIAdjCreateNonemptySubcommMat<br class="">subroutine MatSeqAIJCUSPARSESetGenerateTranspose<br class="">subroutine MatMPIAIJSetUseScalableIncreaseOverlap<br class="">subroutine MatFactorSolveSchurComplementTranspose</font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE.<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE.<br class="">Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem.<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this?<span class="Apple-converted-space"> </span><br class="">Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler?<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">I'll package this up and send it to the IBM XL compiler development team for their examination and comment.<span class="Apple-converted-space"> </span></font><font size="2" class=""><br class=""></font><font size="2" class=""><br class="">Best Regards,<br class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab<br class="">email:<span class="Apple-converted-space"> </span></font><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">roymuss@us.ibm.com</font></u></a><font size="2" class=""><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><font size="2" class=""><br class=""><br class=""><graycol.gif></font><font size="2" color="#424282" class="">Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W</font><font size="2" class=""><br class=""></font><font size="2" color="#5F5F5F" class=""><br class="">From:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">Jacob Faibussowitsch <</font><a href="mailto:faibuss2@illinois.edu" class=""><u class=""><font size="2" color="#0000FF" class="">faibuss2@illinois.edu</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">To:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">Roy Musselman <</font><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">roymuss@us.ibm.com</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">Cc:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">"Gyllenhaal, John C." <</font><a href="mailto:gyllenhaal1@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" class="">gyllenhaal1@llnl.gov</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">Date:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">02/18/2021 02:17 PM</font><font size="2" color="#5F5F5F" class=""><br class="">Subject:<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></font><font size="2" class="">[EXTERNAL] Re: xlf90_r Internal Compiler Error</font></p><div class=""><br class="webkit-block-placeholder"></div><hr width="100%" size="2" align="left" noshade="" style="" class=""><br class=""><br class=""><font size="1" color="#FFFFFF" class=""><br class="">The most recently built version available on the CORAL systems...<span class="Apple-converted-space"> </span></font><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td width="100%" bgcolor="#9CA3A7" valign="middle" class=""><table class="pfptMainWrapper" width="100%" border="0" cellspacing="0" cellpadding="0"><tbody class=""><tr valign="top" class=""><td width="100%" class=""><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td class="pfptTableColumnLeft" width="100%" bgcolor="#9CA3A7"><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td width="100%" valign="middle" class=""><ul class=""><ul class=""><b class=""><font size="4" face="Arial" class="">This Message Is From an External Sender</font></b></ul></ul></td></tr><tr valign="top" class=""><td width="100%" valign="middle" class=""><ul class=""><ul class=""><font size="2" face="Arial" class="">This message came from outside your organization.</font></ul></ul></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><ul class=""><ul class=""><ul class=""><ul class=""><font size="2" class="">The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you?</font></ul></ul></ul></ul><font size="2" class="">I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here.</font><ul class=""><ul class=""><ul class=""><ul class=""><font size="2" class="">The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error.<span class="Apple-converted-space"> </span><br class="">I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing.</font></ul></ul></ul></ul><font size="2" class="">Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run:</font><br class=""><font size="2" class=""><br class="">$ python3 ./reconfigure-arch-linux-c-debug.py</font><br class=""><font size="2" class=""><br class="">It should work. If not, you will have to<span class="Apple-converted-space"> </span></font><br class=""><font size="2" class=""><br class="">$ ./configure —all-the-args —in-the-reconfigure —file</font><br class=""><font size="2" class=""><br class="">Best regards,</font><br class=""><font size="2" class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM]<span class="Apple-converted-space"> </span></font><ul class=""><ul class=""><ul class=""><ul class=""><font size="2" class="">On Feb 18, 2021, at 15:07, Roy Musselman <</font><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">roymuss@us.ibm.com</font></u></a><font size="2" class="">> wrote:</font><p class=""><font size="2" class="">Hi Jacob,<br class=""><br class="">The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you?<br class="">The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error.<span class="Apple-converted-space"> </span><br class="">I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing.<br class=""><br class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab</font><u class=""><font size="2" color="#0000FF" class=""><br class=""></font></u><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">email: roymuss@us.ibm.com</font></u></a><font size="2" class=""><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><font size="2" class=""><br class=""><br class=""><graycol.gif></font><font size="2" color="#424282" class="">Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman</font><font size="2" color="#5F5F5F" class=""><br class=""><br class="">From:<span class="Apple-converted-space"> </span></font><font size="2" class="">Roy Musselman/Rochester/Contr/IBM</font><font size="2" color="#5F5F5F" class=""><br class="">To:<span class="Apple-converted-space"> </span></font><font size="2" class="">LC Hotline <</font><a href="mailto:lc-hotline@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" class="">lc-hotline@llnl.gov</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">Cc:<span class="Apple-converted-space"> </span></font><font size="2" class="">"Gyllenhaal, John C." <</font><a href="mailto:gyllenhaal1@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" class="">gyllenhaal1@llnl.gov</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">Date:<span class="Apple-converted-space"> </span></font><font size="2" class="">02/18/2021 11:18 AM</font><font size="2" color="#5F5F5F" class=""><br class="">Subject:<span class="Apple-converted-space"> </span></font><font size="2" class="">Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error</font></p><p class=""><br class=""></p><hr width="100%" size="2" align="left" noshade="" style="" class=""><font size="2" class=""><br class=""><br class="">I'll take a look.<span class="Apple-converted-space"> </span><br class="">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br class="">Roy Musselman<br class="">IBM HPC Application Analyst at Lawrence Livermore National Lab</font><u class=""><font size="2" color="#0000FF" class=""><br class=""></font></u><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">email: roymuss@us.ibm.com</font></u></a><font size="2" class=""><br class="">LLNL office: 925-422-6033<br class="">Cell: 507-358-8895, Home: 507-281-9565</font><font size="2" class=""><br class=""><br class=""><br class=""><graycol.gif></font><font size="2" color="#424282" class="">LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with</font><font size="2" color="#5F5F5F" class=""><br class=""><br class="">From:<span class="Apple-converted-space"> </span></font><font size="2" class="">LC Hotline <</font><a href="mailto:lc-hotline@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" class="">lc-hotline@llnl.gov</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">To:<span class="Apple-converted-space"> </span></font><font size="2" class="">"Gyllenhaal, John C." <</font><a href="mailto:gyllenhaal1@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" class="">gyllenhaal1@llnl.gov</font></u></a><font size="2" class="">>, Roy Musselman <</font><a href="mailto:roymuss@us.ibm.com" class=""><u class=""><font size="2" color="#0000FF" class="">roymuss@us.ibm.com</font></u></a><font size="2" class="">></font><font size="2" color="#5F5F5F" class=""><br class="">Date:<span class="Apple-converted-space"> </span></font><font size="2" class="">02/18/2021 11:03 AM</font><font size="2" color="#5F5F5F" class=""><br class="">Subject:<span class="Apple-converted-space"> </span></font><font size="2" class="">[EXTERNAL] FW: xlf90_r Internal Compiler Error</font><font size="2" class=""><br class=""></font><hr width="100%" size="2" align="left" noshade="" style="" class=""><font size="2" class=""><br class=""></font><font size="1" color="#FFFFFF" class=""><br class=""><br class="">Hi John, Roy, Can you help this user with the problem that he is...<span class="Apple-converted-space"> </span></font><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td width="100%" bgcolor="#9CA3A7" valign="middle" class=""><table class="pfptMainWrapper" width="100%" border="0" cellspacing="0" cellpadding="0"><tbody class=""><tr valign="top" class=""><td width="100%" class=""><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td class="pfptTableColumnLeft" width="100%" bgcolor="#9CA3A7"><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr valign="top" class=""><td width="100%" valign="middle" class=""><ul class=""><ul class=""><ul class=""><ul class=""><b class=""><font size="4" face="Arial" class="">This Message Is From an External Sender</font></b></ul></ul></ul></ul></td></tr><tr valign="top" class=""><td width="100%" valign="middle" class=""><ul class=""><ul class=""><ul class=""><ul class=""><font size="2" face="Arial" class="">This message came from outside your organization.</font></ul></ul></ul></ul></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><font size="2" face="Calibri" class="">Hi John, Roy,<br class=""><br class="">Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen?<br class=""><br class="">Thanks,<br class="">Ryan<br class="">--<br class="">LC Hotline</font><b class=""><font size="2" face="Calibri" class=""><br class=""><br class="">From:<span class="Apple-converted-space"> </span></font></b><font size="2" face="Calibri" class="">Jacob Faibussowitsch <</font><a href="mailto:faibuss2@illinois.edu" class=""><u class=""><font size="2" color="#0000FF" face="Calibri" class="">faibuss2@illinois.edu</font></u></a><font size="2" face="Calibri" class="">></font><b class=""><font size="2" face="Calibri" class=""><br class="">Date:<span class="Apple-converted-space"> </span></font></b><font size="2" face="Calibri" class="">Wednesday, February 17, 2021 at 5:27 PM</font><b class=""><font size="2" face="Calibri" class=""><br class="">To:<span class="Apple-converted-space"> </span></font></b><font size="2" face="Calibri" class="">LC Hotline <</font><a href="mailto:lc-hotline@llnl.gov" class=""><u class=""><font size="2" color="#0000FF" face="Calibri" class="">lc-hotline@llnl.gov</font></u></a><font size="2" face="Calibri" class="">></font><b class=""><font size="2" face="Calibri" class=""><br class="">Subject:<span class="Apple-converted-space"> </span></font></b><font size="2" face="Calibri" class="">xlf90_r Internal Compiler Error<br class=""><br class="">Hello LC Support,<span class="Apple-converted-space"> </span><br class=""><br class="">While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error:<br class=""><br class="">*** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 ***<br class=""><br class="">I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send.<br class=""><br class="">P.S. My list of loaded modules:<br class=""><br class="">Currently Loaded Modules:<br class="">1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1<br class="">2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12<br class="">3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0<br class=""><br class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM]<span class="Apple-converted-space"> </span></font></ul></ul></ul></ul></ul></ul></div></blockquote></div><br class=""></div></body></html>