<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>More like smaller pieces that need to be combined. Combining them (merging) means sharing the actual data across a sibling communicator and doing some linear algebra to compute the merged matrices (it involves computing a Schur complement of a combined system from the sibling matrices).</div><div><br></div><div>The solver is based on the Hierarchical Poincaré-Steklov (HPS) method, a direct method for solving elliptic PDEs. I had a conversation with Richard at this year’s ATPESC2023 about this idea.<br></div><div><br></div><div>For some more context, here’s the test routine I wrote based on the MatCreateSubMatrix idea. The actual implementation would be part of a recursive merge up a quadtree. Each node's communicator would be a sub-communicator of its parent, and so on. I want to spread the data and compute across any ranks that are involved in that node’s merging. The sizes involved start “small” at each leaf node (say, no more than 256x256), then are essentially doubled up the tree to the root node.</div><div><br></div><div>```</div><div><div style="color: rgb(15, 15, 8); background-color: rgb(255, 255, 255); font-family: Menlo, Monaco, "Courier New", monospace; font-size: 12px; line-height: 18px; white-space: pre;"><div><span style="color: rgb(29, 173, 200); font-style: italic;">void</span> <span style="color: rgb(163, 238, 13);">TEST_petsc_matrix_comm</span>() {</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Create local matrices on local communicator</span></div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> M_local <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">4</span>;</div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> N_local <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">4</span>;</div><div> Mat mat_local;</div><div> <span style="color: rgb(163, 238, 13);">MatCreate</span>(MPI_COMM_SELF, <span style="color: rgb(255, 8, 103);">&</span>mat_local);<span style="color: rgb(144, 144, 144); font-style: italic;"> // Note the MPI_COMM_SELF as a substitute for a sub-communicator of MPI_COMM_WORLD</span></div><div> <span style="color: rgb(163, 238, 13);">MatSetSizes</span>(mat_local, PETSC_DECIDE, PETSC_DECIDE, M_local, N_local);</div><div> <span style="color: rgb(163, 238, 13);">MatSetFromOptions</span>(mat_local);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Set values in local matrix</span></div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span><span style="color: rgb(255, 8, 103);">*</span> row_indices <span style="color: rgb(255, 8, 103);">=</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span><span style="color: rgb(255, 8, 103);">*</span>) <span style="color: rgb(163, 238, 13);">malloc</span>(M_local<span style="color: rgb(255, 8, 103);">*sizeof</span>(<span style="color: rgb(29, 173, 200); font-style: italic;">int</span>));</div><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> i <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; i <span style="color: rgb(255, 8, 103);"><</span> M_local; i<span style="color: rgb(255, 8, 103);">++</span>) {</div><div> row_indices[i] <span style="color: rgb(255, 8, 103);">=</span> i;</div><div> }</div><br><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span><span style="color: rgb(255, 8, 103);">*</span> col_indices <span style="color: rgb(255, 8, 103);">=</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span><span style="color: rgb(255, 8, 103);">*</span>) <span style="color: rgb(163, 238, 13);">malloc</span>(N_local<span style="color: rgb(255, 8, 103);">*sizeof</span>(<span style="color: rgb(29, 173, 200); font-style: italic;">int</span>));</div><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> j <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; j <span style="color: rgb(255, 8, 103);"><</span> M_local; j<span style="color: rgb(255, 8, 103);">++</span>) {;</div><div> col_indices[j] <span style="color: rgb(255, 8, 103);">=</span> j;</div><div> }</div><br><div> <span style="color: rgb(29, 173, 200); font-style: italic;">double</span><span style="color: rgb(255, 8, 103);">*</span> values <span style="color: rgb(255, 8, 103);">=</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">double</span><span style="color: rgb(255, 8, 103);">*</span>) <span style="color: rgb(163, 238, 13);">malloc</span>(M_local<span style="color: rgb(255, 8, 103);">*</span>N_local<span style="color: rgb(255, 8, 103);">*sizeof</span>(<span style="color: rgb(29, 173, 200); font-style: italic;">double</span>));</div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> v <span style="color: rgb(255, 8, 103);">=</span> M_local<span style="color: rgb(255, 8, 103);">*</span>N_local<span style="color: rgb(255, 8, 103);">*</span>rank;</div><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> j <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; j <span style="color: rgb(255, 8, 103);"><</span> N_local; j<span style="color: rgb(255, 8, 103);">++</span>) {</div><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> i <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; i <span style="color: rgb(255, 8, 103);"><</span> M_local; i<span style="color: rgb(255, 8, 103);">++</span>) {</div><div> values[i <span style="color: rgb(255, 8, 103);">+</span> j<span style="color: rgb(255, 8, 103);">*</span>N_local] <span style="color: rgb(255, 8, 103);">=</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">double</span>) v;</div><div> v<span style="color: rgb(255, 8, 103);">++</span>;</div><div> }</div><div> }</div><div> <span style="color: rgb(163, 238, 13);">MatSetValues</span>(mat_local, M_local, row_indices, N_local, col_indices, values, INSERT_VALUES);</div><div> <span style="color: rgb(163, 238, 13);">MatAssemblyBegin</span>(mat_local, MAT_FINAL_ASSEMBLY);</div><div> <span style="color: rgb(163, 238, 13);">MatAssemblyEnd</span>(mat_local, MAT_FINAL_ASSEMBLY);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Create local matrices on global communicator</span></div><div> Mat mat_global;</div><div> IS is_row;</div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> idx[<span style="color: rgb(169, 11, 156);">4</span>] <span style="color: rgb(255, 8, 103);">=</span> {<span style="color: rgb(169, 11, 156);">0</span>, <span style="color: rgb(169, 11, 156);">1</span>, <span style="color: rgb(169, 11, 156);">2</span>, <span style="color: rgb(169, 11, 156);">3</span>};</div><div> <span style="color: rgb(163, 238, 13);">ISCreateGeneral</span>(MPI_COMM_WORLD, M_local, idx, PETSC_COPY_VALUES, <span style="color: rgb(255, 8, 103);">&</span>is_row);</div><div> <span style="color: rgb(163, 238, 13);">MatCreateSubMatrix</span>(mat_local, is_row, <span style="color: rgb(163, 238, 13);">NULL</span>, MAT_INITIAL_MATRIX, <span style="color: rgb(255, 8, 103);">&</span>mat_global);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // View each local mat on global communicator (sleep for `rank` seconds so output is ordered)</span></div><div> <span style="color: rgb(163, 238, 13);">sleep</span>(rank);</div><div> <span style="color: rgb(163, 238, 13);">MatView</span>(mat_global, <span style="color: rgb(169, 11, 156);">0</span>);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Create merged mat on global communicator</span></div><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // For this test, I just put the four locally computed matrices on the diagonal of the merged matrix</span></div><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // In the 4-to-1 merge, this would compute T_merged from T_alpha, T_beta, T_gamma, and T_omega (children)</span></div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> M_merged <span style="color: rgb(255, 8, 103);">=</span> M_local<span style="color: rgb(255, 8, 103);">*</span>size;</div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">int</span> N_merged <span style="color: rgb(255, 8, 103);">=</span> N_local<span style="color: rgb(255, 8, 103);">*</span>size;</div><div> Mat mat_merged;</div><div> <span style="color: rgb(163, 238, 13);">MatCreate</span>(MPI_COMM_WORLD, <span style="color: rgb(255, 8, 103);">&</span>mat_merged);</div><div> <span style="color: rgb(163, 238, 13);">MatSetSizes</span>(mat_merged, PETSC_DECIDE, PETSC_DECIDE, M_merged, N_merged);</div><div> <span style="color: rgb(163, 238, 13);">MatSetFromOptions</span>(mat_merged);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Get values of local matrix to put on diagonal</span></div><div> <span style="color: rgb(29, 173, 200); font-style: italic;">double</span><span style="color: rgb(255, 8, 103);">*</span> values_diag <span style="color: rgb(255, 8, 103);">=</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">double</span><span style="color: rgb(255, 8, 103);">*</span>) <span style="color: rgb(163, 238, 13);">malloc</span>(M_local<span style="color: rgb(255, 8, 103);">*</span>N_local<span style="color: rgb(255, 8, 103);">*sizeof</span>(<span style="color: rgb(29, 173, 200); font-style: italic;">double</span>));</div><div> <span style="color: rgb(163, 238, 13);">MatGetValues</span>(mat_global, M_local, row_indices, N_local, col_indices, values_diag);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Put local matrix contributions into merged matrix (placeholder for computing merged matrix)</span></div><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> i <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; i <span style="color: rgb(255, 8, 103);"><</span> M_local; i<span style="color: rgb(255, 8, 103);">++</span>) {</div><div> row_indices[i] <span style="color: rgb(255, 8, 103);">=</span> i <span style="color: rgb(255, 8, 103);">+</span> M_local<span style="color: rgb(255, 8, 103);">*</span>rank;</div><div> }</div><br><div> <span style="color: rgb(255, 8, 103);">for</span> (<span style="color: rgb(29, 173, 200); font-style: italic;">int</span> j <span style="color: rgb(255, 8, 103);">=</span> <span style="color: rgb(169, 11, 156);">0</span>; j <span style="color: rgb(255, 8, 103);"><</span> N_local; j<span style="color: rgb(255, 8, 103);">++</span>) {</div><div> col_indices[j] <span style="color: rgb(255, 8, 103);">=</span> j <span style="color: rgb(255, 8, 103);">+</span> N_local<span style="color: rgb(255, 8, 103);">*</span>rank;</div><div> }</div><div> <span style="color: rgb(163, 238, 13);">MatSetValues</span>(mat_merged, M_local, row_indices, N_local, col_indices, values_diag, INSERT_VALUES);</div><div> <span style="color: rgb(163, 238, 13);">MatAssemblyBegin</span>(mat_merged, MAT_FINAL_ASSEMBLY);</div><div> <span style="color: rgb(163, 238, 13);">MatAssemblyEnd</span>(mat_merged, MAT_FINAL_ASSEMBLY);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // View merged mat on global communicator</span></div><div> <span style="color: rgb(163, 238, 13);">sleep</span>(rank);</div><div> <span style="color: rgb(163, 238, 13);">MatView</span>(mat_merged, <span style="color: rgb(169, 11, 156);">0</span>);</div><br><div><span style="color: rgb(144, 144, 144); font-style: italic;"> // Clean up</span></div><div> <span style="color: rgb(163, 238, 13);">free</span>(row_indices);</div><div> <span style="color: rgb(163, 238, 13);">free</span>(col_indices);</div><div> <span style="color: rgb(163, 238, 13);">free</span>(values);</div><div> <span style="color: rgb(163, 238, 13);">free</span>(values_diag);</div><div> <span style="color: rgb(163, 238, 13);">MatDestroy</span>(<span style="color: rgb(255, 8, 103);">&</span>mat_local);</div><div> <span style="color: rgb(163, 238, 13);">MatDestroy</span>(<span style="color: rgb(255, 8, 103);">&</span>mat_global);</div><div> <span style="color: rgb(163, 238, 13);">MatDestroy</span>(<span style="color: rgb(255, 8, 103);">&</span>mat_merged);</div><br><div>}</div></div></div><div>```</div><div><br></div><div>With the following output :</div><div><br></div><div>```</div><div><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">(base) </span><span style="font-variant-ligatures: no-common-ligatures; color: #39c026"><b>➜ </b></span><span style="font-variant-ligatures: no-common-ligatures; color: #38b9c7"><b>mpi</b></span><span style="font-variant-ligatures: no-common-ligatures"> </span><span style="font-variant-ligatures: no-common-ligatures; color: #5620f4"><b>git:(</b></span><span style="font-variant-ligatures: no-common-ligatures; color: #ca3323"><b>feature-parallel</b></span><span style="font-variant-ligatures: no-common-ligatures; color: #5620f4"><b>) </b></span><span style="font-variant-ligatures: no-common-ligatures; color: #aaab25"><b>✗</b></span><span style="font-variant-ligatures: no-common-ligatures"> mpirun -n 4 ./mpi_matrix</span></p></div><div><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">Mat Object: 1 MPI process</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 0: (0, 0.) (1, 1.) (2, 2.) (3, 3.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 1: (0, 4.) (1, 5.) (2, 6.) (3, 7.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 2: (0, 8.) (1, 9.) (2, 10.) (3, 11.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 3: (0, 12.) (1, 13.) (2, 14.) (3, 15.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">Mat Object: 1 MPI process</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 0: (0, 16.) (1, 17.) (2, 18.) (3, 19.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 1: (0, 20.) (1, 21.) (2, 22.) (3, 23.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 2: (0, 24.) (1, 25.) (2, 26.) (3, 27.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 3: (0, 28.) (1, 29.) (2, 30.) (3, 31.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">Mat Object: 1 MPI process</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 0: (0, 32.) (1, 33.) (2, 34.) (3, 35.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 1: (0, 36.) (1, 37.) (2, 38.) (3, 39.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 2: (0, 40.) (1, 41.) (2, 42.) (3, 43.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 3: (0, 44.) (1, 45.) (2, 46.) (3, 47.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">Mat Object: 1 MPI process</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures"> type: seqaij</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 0: (0, 48.) (1, 49.) (2, 50.) (3, 51.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 1: (0, 52.) (1, 53.) (2, 54.) (3, 55.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 2: (0, 56.) (1, 57.) (2, 58.) (3, 59.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 3: (0, 60.) (1, 61.) (2, 62.) (3, 63.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">Mat Object: 4 MPI processes</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures"> type: mpiaij</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 0: (0, 0.) (1, 1.) (2, 2.) (3, 3.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 1: (0, 4.) (1, 5.) (2, 6.) (3, 7.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 2: (0, 8.) (1, 9.) (2, 10.) (3, 11.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 3: (0, 12.) (1, 13.) (2, 14.) (3, 15.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 4: (4, 16.) (5, 17.) (6, 18.) (7, 19.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 5: (4, 20.) (5, 21.) (6, 22.) (7, 23.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 6: (4, 24.) (5, 25.) (6, 26.) (7, 27.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 7: (4, 28.) (5, 29.) (6, 30.) (7, 31.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 8: (8, 32.) (9, 33.) (10, 34.) (11, 35.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 9: (8, 36.) (9, 37.) (10, 38.) (11, 39.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 10: (8, 40.) (9, 41.) (10, 42.) (11, 43.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 11: (8, 44.) (9, 45.) (10, 46.) (11, 47.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 12: (12, 48.) (13, 49.) (14, 50.) (15, 51.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 13: (12, 52.) (13, 53.) (14, 54.) (15, 55.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 14: (12, 56.) (13, 57.) (14, 58.) (15, 59.) </span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(0, 0, 0);"><span style="font-variant-ligatures: no-common-ligatures">row 15: (12, 60.) (13, 61.) (14, 62.) (15, 63.) </span></p></div><div>```</div><br id="lineBreakAtBeginningOfMessage"><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>-Damyn</div></div>
</div>
<div><br><blockquote type="cite"><div>On Oct 25, 2023, at 1:47 PM, Barry Smith <bsmith@petsc.dev> wrote:</div><br class="Apple-interchange-newline"><div><div><br> If the matrices are stored as dense it is likely new code is the best way to go. <br><br> What pieces live on the sub communicator? Is it an m by N matrix where m is the number of rows (on that rank) and N is the total number of columns in the final matrix? Or are they smaller "chunks" that need to be combined together?<br><br> Barry<br><br><br><blockquote type="cite">On Oct 25, 2023, at 3:38 PM, Damyn Chipman <damynchipman@u.boisestate.edu> wrote:<br><br>Great thanks, that seemed to work well. This is something my algorithm will do fairly often (“elevating” a node’s communicator to a communicator that includes siblings). The matrices formed are dense but low rank. With MatCreateSubMatrix, it appears I do a lot of copying from one Mat to another. Is there a way to do it with array copying or pointer movement instead of copying entries?<br><br>-Damyn<br><br><blockquote type="cite">On Oct 24, 2023, at 9:51 PM, Jed Brown <jed@jedbrown.org> wrote:<br><br>You can place it in a parallel Mat (that has rows or columns on only one rank or a subset of ranks) and then MatCreateSubMatrix with all new rows/columns on a different rank or subset of ranks.<br><br>That said, you usually have a function that assembles the matrix and you can just call that on the new communicator.<br><br>Damyn Chipman <damynchipman@u.boisestate.edu> writes:<br><br><blockquote type="cite">Hi PETSc developers,<br><br>In short, my question is this: Does PETSc provide a way to move or copy an object (say a Mat) from one communicator to another?<br><br>The more detailed scenario is this: I’m working on a linear algebra solver on quadtree meshes (i.e., p4est). I use communicator subsets in order to facilitate communication between siblings or nearby neighbors. When performing linear algebra across siblings (a group of 4), I need to copy a node’s data (i.e., a Mat object) from a sibling’s communicator to the communicator that includes the four siblings. From what I can tell, I can only copy a PETSc object onto the same communicator.<br><br>My current approach will be to copy the raw data from the Mat on one communicator to a new Mat on the new communicator, but I wanted to see if there is a more “elegant” approach within PETSc.<br><br>Thanks in advance,<br><br>Damyn Chipman<br>Boise State University<br>PhD Candidate<br>Computational Sciences and Engineering<br>damynchipman@u.boisestate.edu<br></blockquote></blockquote><br></blockquote><br></div></div></blockquote></div><br></body></html>