<div dir="ltr"><div dir="ltr">On Thu, Apr 7, 2022 at 6:12 AM Gabriela Nečasová <<a href="mailto:necaga@gmail.com">necaga@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div>Dear PETSc team,</div><div><br></div><div>I would like to ask you a question about the matrix preallocation. <br></div><div>I am using the routine MatMPIAIJSetPreallocation(). <br></div><div><br></div><div>Example: The matrix A has the size 18 x 18 with 168 nonzeros:<br></div><div>A = <br></div><div> 106.21 -91.667 26.042 0 0 0 0 0 0 -2704.2 2862.5 -1354.2 260.42 0 0 0 0 0<br> -91.667 132.25 -91.667 26.042 0 0 0 0 0 2862.5 -4058.3 3122.9 -1354.2 260.42 0 0 0 0<br> 26.042 -91.667 132.25 -91.667 26.042 0 0 0 0 -1354.2 3122.9 -4058.3 3122.9 -1354.2 260.42 0 0 0<br> 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0 260.42 -1354.2 3122.9 -4058.3 3122.9 -1354.2 260.42 0 0<br> 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0 260.42 -1354.2 3122.9 -4058.3 3122.9 -1354.2 260.42 0<br> 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0 260.42 -1354.2 3122.9 -4058.3 3122.9 -1354.2 260.42<br> 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0 260.42 -1354.2 3122.9 -4058.3 3122.9 -1354.2<br> 0 0 0 0 0 26.042 -91.667 132.25 -91.667 0 0 0 0 260.42 -1354.2 3122.9 -4058.3 2862.5<br> 0 0 0 0 0 0 26.042 -91.667 106.21 0 0 0 0 0 260.42 -1354.2 2862.5 -2704.2<br> 9.3542 -8.3333 2.6042 0 0 0 0 0 0 106.21 -91.667 26.042 0 0 0 0 0 0<br> -8.3333 11.958 -8.3333 2.6042 0 0 0 0 0 -91.667 132.25 -91.667 26.042 0 0 0 0 0<br> 2.6042 -8.3333 11.958 -8.3333 2.6042 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0 0<br> 0 2.6042 -8.3333 11.958 -8.3333 2.6042 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0 0<br> 0 0 2.6042 -8.3333 11.958 -8.3333 2.6042 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0 0<br> 0 0 0 2.6042 -8.3333 11.958 -8.3333 2.6042 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042 0<br> 0 0 0 0 2.6042 -8.3333 11.958 -8.3333 2.6042 0 0 0 0 26.042 -91.667 132.25 -91.667 26.042<br> 0 0 0 0 0 2.6042 -8.3333 11.958 -8.3333 0 0 0 0 0 26.042 -91.667 132.25 -91.667<br> 0 0 0 0 0 0 2.6042 -8.3333 9.3542 0 0 0 0 0 0 26.042 -91.667 106.21<br></div><div><br></div><div>I am wondering how the d_nz and o_nz values should be set.</div><div>I read the docs: <a href="https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html</a></div><div><br></div><div><b>For example, for 2 processes P1, P2: </b><br></div><div>mi - denote the number of rows which belong to process Pi<br></div><div>P0: m0 = 9<br></div><div>P1: m1 = 9 <br><br></div><div>The dn_nz and o_nz should be: <br></div><div>P0: d_nz = 5; o_nz = 7<br></div><div>P1: d_nz = 5; o_nz = 7<br><div><br></div><div>
Therefore, we are allocating m*(d_nz+o_nz) storage locations for every process:<br></div><div>P0: 9*(5+7) = 108</div><div>
P1: 9*(5+7) = 108
<br></div><div>So, in total, we have 216 storage locations to store 168 nonzeros. <br></div><div><br></div><div>But I am doing more experiments and I use the various number of processes - e.g. 2, 4, 8, 19, 32, etc. <br><br></div><div><b>For example, if we take 3 processes - P0, P1, P2:</b><br>
<div>P0: m0 = 6<br></div><div>P1: m1 = 6</div><div>P2: m2 = 6</div><div><br></div><div>
<div>The dn_nz and o_nz should be: <br></div><div>P0: d_nz = 5; o_nz = 7
(I took the maximum number of o_nz)
</div><div>P1: d_nz = 3; o_nz = 8 (I took the maximum number of o_nz)<br></div><div>P2: d_nz = 5; o_nz = 7
(I took the maximum number of o_nz)
</div><div><div><br></div><div>
Therefore, we are allocating m*(d_nz+o_nz) storage locations for every process:<br></div><div>P0: 6*(5+7) = 72<br></div><div>
P1: 6*(3+8) = 66
<br></div><div>P2: 6*(5+7) = 72<br></div><div>So, in total, we have 210 storage locations to store 168 nonzeros. <br></div><div><br></div><div><b>My questions are:</b> <br></div><div>1) How should I set the d_nz and o_nz values when the number of processes changes in every experiment? <br></div><div>-- I suppose, there has to be some "general" settings, so you do not have to change these values for every experiment, right? <br></div><div>
-- For better illustration, I am sending you figures of the matrix A in the attachment.<br></div></div></div></div></div></div></blockquote><div><br></div><div>I think the correct answer is that d_nz/o_nz should only be used for testing or example codes. Production codes should</div><div>specify the number of nonzeros in every row, d_nnz/o_nnz. We provide support for doing this easily with the</div><div>MatPreallocator class..</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div></div><div>2) If I run the code sequentially, then the
MatMPIAIJSetPreallocation() routine will crash?</div></div></div></div></div></div></blockquote><div><br></div><div>If you run sequentially, meaning with the type MATSEQAIJ, then this function is ignored, so your matrix</div><div>will not be preallocated. This can still work, but will be slow.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>3) If I need to generate a large identity matrix, is this a correct approach, please? <br>//create matrix I (diagonal matrix)<br> ierr = MatCreate(comm, &Ione);CHKERRQ(ierr);<br> ierr = MatSetType(Ione, MATMPIAIJ);CHKERRQ(ierr);<br> ierr = MatSetSizes(Ione, PETSC_DECIDE, PETSC_DECIDE, data_size, data_size);CHKERRQ(ierr);<br><br> // parallel sparse - preallocation for better performance<br> ierr = MatMPIAIJSetPreallocation(Ione, 1, PETSC_NULL, 0, PETSC_NULL);CHKERRQ(ierr);<br><br> // create vector with ones<br> ierr = VecCreate(PETSC_COMM_WORLD, &diagVec);CHKERRQ(ierr);<br> ierr = VecSetType(diagVec, VECMPI);CHKERRQ(ierr);<br> ierr = VecSetSizes(diagVec, PETSC_DECIDE, data_size);CHKERRQ(ierr);<br> ierr = VecSet(diagVec, 1);CHKERRQ(ierr);<br> // set the diagonal of matrix I using the vector diagVec<br> ierr = MatDiagonalSet(Ione, diagVec, INSERT_VALUES);CHKERRQ(ierr);<br><br> ierr = MatAssemblyBegin(Ione, MAT_FINAL_ASSEMBLY);<br> ierr = MatAssemblyEnd(Ione, MAT_FINAL_ASSEMBLY);</div></div></div></div></div></div></blockquote><div><br></div><div>Yes, that is a fine way to do it.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Thank you very much in advance. <br></div><div>Kind regards<br></div><div>Gabi<br></div><div><br></div></div>
</div>
</div><div> <br></div>
</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>