<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 28 November 2016 at 10:43, Rolf Kuiper <span dir="ltr"><<a href="mailto:kuiper@mpia.de" target="_blank">kuiper@mpia.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Hi Dave,<div><br></div><div>Thanks a lot for your prompt reply! This is even easier than I thought (and that is most likely the reason, why I could not think into this direction) :)</div><div>And yes, I might/should upgrade our PETSc version used (3.1), but it will take me some days to check the full code package.</div><div><br></div><div>Now, I will create my DA (now in 3.6 called DMDA) for the coupled system via</div><div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">DACreate3d(</span>PETSC_COMM_WORLD, <wbr>Periodicity, DA_STENCIL_BOX, <span style="font-variant-ligatures:no-common-ligatures">Nx,</span><span style="font-variant-ligatures:no-common-ligatures"> Ny, Nz, Px, Py, Pz, 4, 1, </span>lx, ly, lz, &MyDA);</div></div><div>instead of</div><div><div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">DACreate3d(</span>PETSC_COMM_WORLD, <wbr>Periodicity, DA_STENCIL_BOX, <span style="font-variant-ligatures:no-common-ligatures">Nx,</span><span style="font-variant-ligatures:no-common-ligatures"> Ny, Nz, Px, Py, Pz, 1, 1, </span>lx, ly, lz, &MyDA);</div></div></div><div>which I have used for the single equation.</div><div><br></div><div>I would like to ask one follow-up question:</div><div>Currently, I loop over my matrix (and vectors) via three for-loops over the 3 spatial directions, and set the columns indices in the following way:</div><div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">for(k){</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"> row.k = k;</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">0</span><span style="font-variant-ligatures:no-common-ligatures">].k = k;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">].k = k;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">2</span><span style="font-variant-ligatures:no-common-ligatures">].k = k;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">3</span><span style="font-variant-ligatures:no-common-ligatures">].k = k;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">4</span><span style="font-variant-ligatures:no-common-ligatures">].k = k+</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">5</span><span style="font-variant-ligatures:no-common-ligatures">].k = k-</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> for(j){</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"> row.j = j;</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">0</span><span style="font-variant-ligatures:no-common-ligatures">].j = j;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">].j = j;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">2</span><span style="font-variant-ligatures:no-common-ligatures">].j = j+</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">3</span><span style="font-variant-ligatures:no-common-ligatures">].j = j-</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">4</span><span style="font-variant-ligatures:no-common-ligatures">].j = j;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">5</span><span style="font-variant-ligatures:no-common-ligatures">].j = j;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> for(i){</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"> row.i = i;</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">0</span><span style="font-variant-ligatures:no-common-ligatures">].i = i+</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">].i = i-</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">1</span><span style="font-variant-ligatures:no-common-ligatures">;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">2</span><span style="font-variant-ligatures:no-common-ligatures">].i = i;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">3</span><span style="font-variant-ligatures:no-common-ligatures">].i = i;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">4</span><span style="font-variant-ligatures:no-common-ligatures">].i = i;</span></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> col[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(39,42,216)">5</span><span style="font-variant-ligatures:no-common-ligatures">].i = i;</span></div></div><div>Now for the coupled equations, I should overall additionally loop over the number of DOFs (just 2 in my previous email example). </div></div></blockquote><div><br></div><div>So presumably you are using MatSetValuesStencil() to insert entries into the matrix.<br><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesStencil.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesStencil.html</a><br><br></div><div>You can keep using MatSetValuesStencil(), for you multi-component example, however you will additionally need to enter a value for the member "c" within the MatStencil struct. <br>See <br><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatStencil.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatStencil.html</a><br><br>"c" relates to the component or DOF index. So the code above will need to be modified slightly to define values for row.c and col[0].c , col[1].c ... etc<br><br><br></div><div>Alternatively you can use MatSetValuesStencilBlocked()<br><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesBlockedStencil.html#MatSetValuesBlockedStencil">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesBlockedStencil.html#MatSetValuesBlockedStencil</a><br><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>Could you give me an easy example or pseudo-code for the associated assignment of columns (I have the same stencil in each of the submatrices, so for the 2 DOFs in 3D, I would get 7+6=13 column entries per row)? Or can you link me to an existing example within the PETSc help?</div></div></blockquote><div><br>Note that the manual pages I've sent links to contains links to example codes (see bottom of the webpage) where you can see how to use these functions.<br><br></div><div>Thanks,<br></div><div> Dave<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>Again, Thanks a lot!</div><span class="gmail-HOEnZb"><font color="#888888"><div>Rolf</div></font></span><div><div class="gmail-h5"><div><br></div><div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><br></div></div><div><br></div><div><br><div><blockquote type="cite"><div>Am 24.11.2016 um 22:30 schrieb Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>>:</div><br class="gmail-m_-4428999492873022765Apple-interchange-newline"><div><div style="white-space:pre-wrap">When you create the DMDA, set the number of DOFs (degrees of freedom) per point to be 2 instead of 1. <br><br>You must be using and ancient version of petsc given the function names you quoted. Consider upgrading to 3.7<br><br>Thanks,<br> Dave</div><br><div class="gmail_quote"><div dir="ltr">On Thu, 24 Nov 2016 at 20:24, Rolf Kuiper <<a href="mailto:kuiper@mpia.de" target="_blank">kuiper@mpia.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="gmail-m_-4428999492873022765gmail_msg"><div class="gmail-m_-4428999492873022765gmail_msg">Dear PETSc users,</div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">maybe this is an easy question, but I can’t find the information right away in the user’s guide nor online.</div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">What I am currently doing and which works fine:</div><div class="gmail-m_-4428999492873022765gmail_msg">To solve a partial differential equation for the quantity q on a parallel distributed grid, which is represented by the Distributed Array MyDA, I am currently creating the associated sparse matrix for the KSP solver via</div><div class="gmail-m_-4428999492873022765gmail_msg"><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo" class="gmail-m_-4428999492873022765gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail-m_-4428999492873022765gmail_msg">DAGetMatrix(MyDA, MATMPIAIJ, &MyMatrix);</span></div></div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">The solution vector and right hand side vector are created via</div><div class="gmail-m_-4428999492873022765gmail_msg"><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo" class="gmail-m_-4428999492873022765gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail-m_-4428999492873022765gmail_msg">DACreateGlobalVector(MyDA, &MyRightHandSideVector);</span></div></div><div class="gmail-m_-4428999492873022765gmail_msg"><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo" class="gmail-m_-4428999492873022765gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail-m_-4428999492873022765gmail_msg">VecDuplicate(<wbr>MyRightHandSideVector, &GlobalSolutionVector);</span></div></div><div style="margin:0px;font-size:11px;line-height:normal;font-family:menlo" class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">The DA is constructed using <span style="font-family:menlo;font-size:11px;font-variant-ligatures:no-common-ligatures" class="gmail-m_-4428999492873022765gmail_msg">DACreate3d()</span><span style="font-family:menlo;font-size:11px" class="gmail-m_-4428999492873022765gmail_msg"> </span>with the corresponding regular structured grid information. </div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">And here is my problem:</div><div class="gmail-m_-4428999492873022765gmail_msg">Now, I would like to solve a coupled system of equations for the quantities q1 and q2 on the same grid. I.e., the matrix should just get the double number of rows and columns, the vectors contain twice the number of entries (e.g. first all q1s and then all q2s). And I would like to be sure that the entries of q1 and q2, which are associated with the same grid cell are located on the same processor.</div><div class="gmail-m_-4428999492873022765gmail_msg">Is there already a pre-defined structures available (such as <span style="font-family:menlo;font-size:11px" class="gmail-m_-4428999492873022765gmail_msg">MATMPIAIJ</span>) within PETSc to enlarge such a single equation to store the entries of coupled equations? Such as</div><div class="gmail-m_-4428999492873022765gmail_msg"><span style="font-family:menlo;font-size:11px" class="gmail-m_-4428999492873022765gmail_msg">DACreateTwiceTheGlobalVector()</span><wbr>?</div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">The equation is (simplified) of the form</div><div class="gmail-m_-4428999492873022765gmail_msg">d/dt q1 + grad q2 = 0</div><div class="gmail-m_-4428999492873022765gmail_msg">d/dt q2 + f(q1) = 0</div><div class="gmail-m_-4428999492873022765gmail_msg">with an arbitrary function f depending on q1.</div><div class="gmail-m_-4428999492873022765gmail_msg"><br class="gmail-m_-4428999492873022765gmail_msg"></div><div class="gmail-m_-4428999492873022765gmail_msg">Thanks a lot for your help in advance,</div><div class="gmail-m_-4428999492873022765gmail_msg">Rolf</div></div></blockquote></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div>