<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Sorry, I didn't notice these emails for a long time. <div class=""><br class=""></div><div class=""> PETSc does provide a "simple" mechanism to redistribute your matrix that does not require you to explicitly do the redistribution. </div><div class=""><br class=""></div><div class=""> You must create a MPIAIJ matrix over all the MPI ranks, but simply provide all the rows on the first rank and zero rows on the rest of the ranks (you can use <a href="https://petsc.org/release/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html#MatCreateMPIAIJWithArrays" class="">MatCreateMPIAIJWithArrays</a>) then use -ksp_type preonly -pc_type redistribute You control the parallel KSP and preconditioner by using for example -redistribute_ksp_type gmres -redistribute_pc_type bjacobi </div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class="">The PC type of redistribute manages distributing the matrix and vectors across all the ranks for you. As the PETSc documentation notes this is not a recommended use of PETSc for large numbers of ranks, due to Amdahl's law; for truly good parallel performance you must build the matrix in parallel.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 8, 2021, at 12:32 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 7, 2021 at 10:04 PM Faraz Hussain <<a href="mailto:faraz_hussain@yahoo.com" class="">faraz_hussain@yahoo.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The matrix in memory is in IJV (Spooles ) or CSR3 ( Pardiso ). The application was written to use a variety of different direct solvers but Spooles and Pardiso are what I am most familiar with. <br class=""></blockquote><div class="">I assume the CSR3 has the a, i, j arrays used in petsc's MATAIJ.</div><div class="">You can create a MPIAIJ matrix A with <a href="https://petsc.org/release/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html#MatCreateMPIAIJWithArrays" class="">MatCreateMPIAIJWithArrays</a>, with only rank 0 providing data (i.e., other ranks just have m=n=0, i=j=a=NULL)</div><div class="">Then you call <a href="https://www.mcs.anl.gov/petsc/petsc-3.7/docs/manualpages/Mat/MatGetSubMatrix.html" class="">MatGetSubMatrix</a>(A,isrow,iscol,reuse,&B) to redistribute the imbalanced A to a balanced matrix B. </div><div class="">You can use PetscLayoutCreate() and friends to create a row map and a column map (as if they are B's) and use them to get ranges of rows/cols each rank wants to own, and then build the isrow, iscol with ISCreateStride()</div><div class=""><br class=""></div><div class="">My approach is kind of verbose. I would let Jed and Matt comment whether there are better ones.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On Tuesday, December 7, 2021, 10:33:24 PM EST, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote: <br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On Tue, Dec 7, 2021 at 9:06 PM Faraz Hussain via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" class="">petsc-users@mcs.anl.gov</a>> wrote:<br class="">
> Thanks, I took a look at ex10.c in ksp/tutorials . It seems to do as you wrote, "it efficiently gets the matrix from the file spread out over all the ranks.".<br class="">
> <br class="">
> However, in my application I only want rank 0 to read and assemble the matrix. I do not want other ranks trying to get the matrix data. The reason is the matrix is already in memory when my application is ready to call the petsc solver.<br class="">
What is the data structure of your matrix in memory?<br class="">
<br class="">
> <br class="">
> <br class="">
> So if I am running with multiple ranks, I don't want all ranks assembling the matrix. This would require a total re-write of my application which is not possible . I realize this may sounds confusing. If so, I'll see if I can create an example that shows the issue.<br class="">
> <br class="">
> <br class="">
> <br class="">
> <br class="">
> <br class="">
> On Tuesday, December 7, 2021, 10:13:17 AM EST, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote: <br class="">
> <br class="">
> <br class="">
> <br class="">
> <br class="">
> <br class="">
> <br class="">
> If you use MatLoad() it never has the entire matrix on a single rank at the same time; it efficiently gets the matrix from the file spread out over all the ranks. <br class="">
> <br class="">
>> On Dec 6, 2021, at 11:04 PM, Faraz Hussain via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" class="">petsc-users@mcs.anl.gov</a>> wrote:<br class="">
>> <br class="">
>> I am studying the examples but it seems all ranks read the full matrix. Is there an MPI example where only rank 0 reads the matrix? <br class="">
>> <br class="">
>> I don't want all ranks to read my input matrix and consume a lot of memory allocating data for the arrays. <br class="">
>> <br class="">
>> I have worked with Intel's cluster sparse solver and their documentation states:<br class="">
>> <br class="">
>> " Most of the input parameters must be set on the master MPI process only, and ignored on other processes. Other MPI processes get all required data from the master MPI process using the MPI communicator, comm. "<br class="">
> <br class="">
> <br class="">
<br class="">
</blockquote></div></div>
</div></blockquote></div><br class=""></div></body></html>