<div dir="ltr">So far, I've tried this:<div><br></div><div><font face="monospace">// fill_in_matrix.cc<br><br>static char help[] = "Fill in a parallel COO format sparse matrix.";<br><br>#include <petsc.h><br>#include <vector><br><br>int main(int argc, char **args)<br>{<br> MPI_Comm comm;<br> Mat A;<br> PetscInt m = 5;<br> PetscMPIInt rank, size;<br><br> PetscCall(PetscInitialize(&argc, &args, NULL, help));<br><br> comm = PETSC_COMM_WORLD;<br> PetscCallMPI(MPI_Comm_rank(comm, &rank));<br> PetscCallMPI(MPI_Comm_size(comm, &size));<br><br> PetscCall(MatCreate(comm, &A));<br> PetscCall(MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, m, m));<br> PetscCall(MatSetFromOptions(A));<br> PetscCall(MatSetUp(A));<br><br> std::vector<PetscInt> coo_i{0, 0, 1, 2, 3, 4};<br> std::vector<PetscInt> coo_j{0, 0, 1, 2, 3, 4};<br> std::vector<PetscScalar> coo_v{2, -1, 2, 3, 4, 5};<br><br> PetscCallMPI(MPI_Comm_rank(comm, &rank));<br><br> if (rank == 0)<br> {<br> for (size_t j = 0; j < coo_i.size(); j++)<br> PetscCall(MatSetValues(A,<br> 1, &<a href="http://coo_i.at">coo_i.at</a>(j),<br> 1, &<a href="http://coo_j.at">coo_j.at</a>(j),<br> &<a href="http://coo_v.at">coo_v.at</a>(j),<br> ADD_VALUES));<br><br> PetscCall(MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY));<br> PetscCall(MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY));<br> PetscCall(MatView(A, PETSC_VIEWER_STDERR_WORLD));<br> }<br><br> PetscCall(MatDestroy(&A));<br> PetscCall(PetscFinalize());<br><br> return 0;<br>}</font><br></div><div><div><font face="monospace"><br></font></div><div><font face="arial, sans-serif">When running with 1 process, it runs like a charm, but when running with more than one process, the code does not finish. Somehow it gets stuck forever.</font></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qua., 21 de jun. de 2023 às 14:12, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Jun 21, 2023 at 12:57 PM Diego Magela Lemos <<a href="mailto:diegomagela@usp.br" target="_blank">diegomagela@usp.br</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Unfortunately, I can modify ex2 to perform the matrix fill in using only one rank, although I have understood how it works.<div>Is it so hard to do that?<br></div></div></blockquote><div><br></div><div>It should not be. Maybe describe what is not clear? ex2 runs in parallel now.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thank you.<br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qua., 21 de jun. de 2023 às 12:20, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">ex2 looks the same as the code at the beginning of the thread, which looks fine to me, yet fails.<br><div>(the only thing I can think of is that &<a href="http://v.at" target="_blank">v.at</a>(i) is not doing what one wants)</div><div><br></div><div>Diego: I would start with this ex2.c, add your view statement, verify; incrementally change ex2 to your syntax and see where it breaks.</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 21, 2023 at 9:50 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Jun 21, 2023 at 9:22 AM Diego Magela Lemos <<a href="mailto:diegomagela@usp.br" target="_blank">diegomagela@usp.br</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Please, could you provide a minimal working example (or link) of how to do this?<br></div></blockquote><div><br></div><div>You can see here</div><div><br></div><div> <a href="https://petsc.org/main/src/ksp/ksp/tutorials/ex2.c.html" target="_blank">https://petsc.org/main/src/ksp/ksp/tutorials/ex2.c.html</a></div><div><br></div><div>that each process only sets values for the rows it owns.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em ter., 20 de jun. de 2023 às 15:08, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Tue, Jun 20, 2023 at 2:02 PM Diego Magela Lemos <<a href="mailto:diegomagela@usp.br" target="_blank">diegomagela@usp.br</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">So... what do I need to do, please?<br>Why am I getting wrong results when solving the linear system if the matrix is filled in with <font face="monospace">MatSetPreallocationCOO</font> and <font face="monospace">MatSetValuesCOO?</font></div></blockquote><div><br></div><div>It appears that you have _all_ processes submit _all_ triples (i, j, v). Each triple can only be submitted by a single process. You can fix this in many ways. For example, an easy but suboptimal way is just to have process 0 submit them all, and all other processes submit nothing.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em ter., 20 de jun. de 2023 às 14:56, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> writes:<br>
<br>
>> The matrix entries are multiplied by 2, that is, the number of processes<br>
>> used to execute the code.<br>
>><br>
><br>
> No. This was mostly intended for GPUs, where there is 1 process. If you<br>
> want to use multiple MPI processes, then each process can only introduce<br>
> some disjoint subset of the values. This is also how MatSetValues() works,<br>
> but it might not be as obvious.<br>
<br>
They need not be disjoint, just sum to the expected values. This interface is very convenient for FE and FV methods. MatSetValues with ADD_VALUES has similar semantics without the intermediate storage, but it forces you to submit one element matrix at a time. Classic parallelism granularity versus memory use tradeoff with MatSetValuesCOO being a clear win on GPUs and more nuanced for CPUs.<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>