<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Apr 20, 2018 at 11:05 AM, Klaus Burkart <span dir="ltr"><<a href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:10px"><div></div>
<div><font size="3"><span>I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc:<br><br>Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process.<br><br>Looking at the manual and <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>petsc-current/docs/<wbr>manualpages/Mat/MatCreateAIJ.<wbr>html#MatCreateAIJ</a>, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays?<br></span></font></div></div></div></blockquote><div><br></div><div>This is SPMD programming, so the call on each process gives a different n, and collectively you have 16 different n values.</div><div><br></div><div>"Using MPI" is a great book for understanding this.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:10px"><div><font size="3"><span> MatCreate(...,&A);<br><br> MatSetType(A,MATMPIAIJ);<br><br> MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?!<br><br> MatMPIAIJSetPreallocation(A,..<wbr>.);<br><br><br>How can I enter the parallel global-local matrix structure?<br><br>How can the memory preallocation be done?<br><br>Klaus</span></font><br></div><div><br></div>
<div id="m_-7136202847276075343yahoo_quoted_5002509784" class="m_-7136202847276075343yahoo_quoted">
<div style="font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;color:#26282a">
<div>
Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> Folgendes geschrieben:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">> On Apr 18, 2018, at 4:42 PM, <a href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a> wrote:<br></div><div dir="ltr">> <br></div><div dir="ltr">> So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core?<br></div><div dir="ltr"><br></div><div dir="ltr"> To really use PETSc efficiently/effectively you need to generate your matrix in parallel.<br></div><div dir="ltr"><br></div><div dir="ltr"> Barry<br></div><div dir="ltr"><br></div><div dir="ltr">> <br></div><div dir="ltr">> Klaus<br></div><div dir="ltr">> <br></div><div dir="ltr">> Von meinem Huawei-Mobiltelefon gesendet<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> -------- Originalnachricht --------<br></div><div dir="ltr">> Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import?<br></div><div dir="ltr">> Von: "Smith, Barry F." <br></div><div dir="ltr">> An: Klaus Burkart <br></div><div dir="ltr">> Cc: PETSc Users List <br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF.<br></div><div dir="ltr">> <br></div><div dir="ltr">> If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0.<br></div><div dir="ltr">> <br></div><div dir="ltr">> Barry<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote:<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > More questions about matrix and vector type selection for my application:<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz?<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > I started with:<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > MatCreate(PETSC_COMM_WORLD, &M);<br></div><div dir="ltr">> > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N);<br></div><div dir="ltr">> > MatSetFromOptions(M);<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. <br></div><div dir="ltr">> > <br></div><div dir="ltr">> > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation?<br></div><div dir="ltr">> > <br></div><div dir="ltr">> > In this case, where would the decompositioning / MPI process allocation take place?<br></div><div dir="ltr">> <br></div></div>
</div>
</div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>