<div dir="ltr"><div dir="ltr">On Tue, Dec 7, 2021 at 10:25 PM Faraz Hussain <<a href="mailto:faraz_hussain@yahoo.com">faraz_hussain@yahoo.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks, that makes sense. I guess I was hoping petsc ksp is like intel's cluster sparse solver where it handles distributing the matrix to the other ranks for you. <br>
<br>
It sounds like that is not the case and I need to manually distribute the matrix to the ranks?<br></blockquote><div><br></div><div>You can call</div><div><br></div><div> <a href="https://petsc.org/main/docs/manualpages/Mat/MatCreateSubMatricesMPI.html">https://petsc.org/main/docs/manualpages/Mat/MatCreateSubMatricesMPI.html</a><br></div><div><br></div><div>to distribute the matrix.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On Tuesday, December 7, 2021, 10:18:04 PM EST, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote: <br>
<br>
<br>
<br>
<br>
<br>
On Tue, Dec 7, 2021 at 10:06 PM Faraz Hussain via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> Thanks, I took a look at ex10.c in ksp/tutorials . It seems to do as you wrote, "it efficiently gets the matrix from the file spread out over all the ranks.".<br>
> <br>
> However, in my application I only want rank 0 to read and assemble the matrix. I do not want other ranks trying to get the matrix data. The reason is the matrix is already in memory when my application is ready to call the petsc solver. <br>
> <br>
> So if I am running with multiple ranks, I don't want all ranks assembling the matrix. This would require a total re-write of my application which is not possible . I realize this may sounds confusing. If so, I'll see if I can create an example that shows the issue.<br>
<br>
MPI is distributed memory parallelism. If we want to use multiple ranks, then parts of the matrix must<br>
be in the different memories of the different processes. If you already assemble your matrix on process<br>
0, then you need to communicate it to the other processes, perhaps using MatGetSubmatrix().<br>
<br>
THanks,<br>
<br>
Matt<br>
<br>
> On Tuesday, December 7, 2021, 10:13:17 AM EST, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote: <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> If you use MatLoad() it never has the entire matrix on a single rank at the same time; it efficiently gets the matrix from the file spread out over all the ranks. <br>
> <br>
>> On Dec 6, 2021, at 11:04 PM, Faraz Hussain via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
>> <br>
>> I am studying the examples but it seems all ranks read the full matrix. Is there an MPI example where only rank 0 reads the matrix? <br>
>> <br>
>> I don't want all ranks to read my input matrix and consume a lot of memory allocating data for the arrays. <br>
>> <br>
>> I have worked with Intel's cluster sparse solver and their documentation states:<br>
>> <br>
>> " Most of the input parameters must be set on the master MPI process only, and ignored on other processes. Other MPI processes get all required data from the master MPI process using the MPI communicator, comm. "<br>
> <br>
> <br>
> <br>
<br>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
<br>
<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>