<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:10.0pt;
font-family:"Calibri","sans-serif";}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-compose;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Dear all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We are using PETSc in our (Fortran) CFD code, which we have recently parallelized.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We need to access the values stored in the solution vector to assemble the matrix of the linear system. In particular, we need to access the values on the local elements, plus the ones on the first neighbors, which might be handled by other
processes.<o:p></o:p></p>
<p class="MsoNormal">Hence, we defined our solution vector as MPI Vec with ghost values.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">At the moment, to access all the required values, at each time step we update the ghost values, then call ‘VecGhostGetLocalForm’ and finally ‘VecGetArrayReadF90’.<o:p></o:p></p>
<p class="MsoNormal">Since all these procedures are collective, we store the gotten values in a local Fortran array (of length n_local + n_ghosts) and then proceed with the parallel matrix assembly.<o:p></o:p></p>
<p class="MsoNormal">The code works perfectly, so I believe we are doing things in the right way.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">However, I was wondering whether there is a more efficient way to access the local values of a ghosted vector; something not collective, so that we could access the values on-demand while assembling the matrix.
<o:p></o:p></p>
<p class="MsoNormal">In this way, we could avoid storing the values in Fortran array, thus saving memory.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Less important issue, but it puzzles me: Why is VecGetArrayReadF90 collective while its C counterpart VecGetArrayRead is not? (same thing for VecRestoreArrayReadF90 and VecRestoreArrayRead).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks a lot for your time and help.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Best regards,<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Marco Tiberga<o:p></o:p></p>
<p class="MsoNormal">PhD candidate<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">Delft University of Technology</span><o:p></o:p></p>
<p class="MsoNormal">Faculty of Applied Sciences<o:p></o:p></p>
<p class="MsoNormal">Radiation Science & Technology Department<o:p></o:p></p>
<p class="MsoNormal">Mekelweg 15, 2629 JB Delft, The Netherlands<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:2.0pt">E-Mail:<span style="color:gray"> </span><a href="mailto:m.tiberga@tudelft.nl">m.tiberga@tudelft.nl</a><o:p></o:p></p>
<p class="MsoNormal"><span lang="NL">Website: </span><a href="http://www.nera.rst.tudelft.nl/" target="_BLANK"><span lang="NL">http://www.nera.rst.tudelft.nl/</span></a><span lang="NL"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL"> <o:p></o:p></span></p>
</div>
</body>
</html>