<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Latha;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I am running NHEP across 16 MPI processors over 16 nodes in a matrix of global size of 1,000,000x1,000,000 with approximately global 16,000,000 non-zeros. Each node has the 1D row distribution of the matrix
with exactly 62500 rows and 1 million columns with 1million non-zeros as CSR/COO matrix.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am generating this graph as follows. It takes approximately 12 seconds to insert 25000 NNZ into petsc matrix with MatSetValues which means it is taking closer to 10 minutes to 1million NNZ’s in every processes.
It takes 12 seconds for assembly. Is these times normal? Is there a faster way of doing it? I am unable to construct matrices of 1 billion global nnz’s in which each process has closer to 100 million entries.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Generate_petsc_matrix(int n_rows, int n_cols, int n_nnz,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PetscInt *row_idx, PetscInt *col_idx, PetscScalar *val,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> const MPICommunicator& communicator) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int *start_row = new int[communicator.size()];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Allgather(&n_rows, 1, MPI_INT, all_proc_rows, 1, MPI_INT, MPI_COMM_WORLD);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> start_row[0] = 0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> for (int i = 0; i < communicator.size(); i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (i > 0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> start_row[i] = start_row[i - 1] + all_proc_rows[i];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> } <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatCreate(PETSC_COMM_WORLD, &A);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatSetType(A, MATMPIAIJ);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatSetSizes(A, n_rows, PETSC_DECIDE, global_rows, n_cols);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatMPIAIJSetPreallocation(A, PETSC_DEFAULT, PETSC_NULL, PETSC_DEFAULT, PETSC_NULL);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PetscInt local_row_idx;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PetscInt local_col_idx;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PetscScalar local_val;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int my_start_row = start_row[MPI_RANK];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int my_start_col = 0; <o:p>
</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">double petsc_insert_time=0.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> for (int i = 0; i < n_nnz; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> local_row_idx = my_start_row + row_idx[i];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> local_col_idx = my_start_col + col_idx[i];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> local_val = val[i];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">tic();<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ierr = MatSetValues(A, 1, &local_row_idx, 1, &local_col_idx, &local_val, INSERT_VALUES);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> petsc_insert_time += toc();<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (i % 25000 == 0){<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PRINTROOT("25000 time::" << petsc_insert_time);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> petsc_insert_time=0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CHKERRV(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PRINTROOT("100000 time::" << petsc_insert_time);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> tic();<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> petsc_insert_time = toc();<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> PRINTROOT("calling assembly to end::took::" << petsc_insert_time);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">Ramki<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>