<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> What is large? If A and B have dimensions of 1000, then the Kronecker product is of size 1,000,000. Do you want the Kronecker product to be explicitly formed or just available as matrix vector products? If just explicitly available then I think you can just store sparse A (for example) completely on all ranks, 10,000 by 10,000 sparse matrix is small for sequential) while B is distributed.<div><br></div><div>Barry</div><div><br><div><br><blockquote type="cite"><div>On Jan 30, 2023, at 2:48 PM, Guglielmo, Tyler Hardy <guglielmo2@llnl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Both matrices (A and B) would be approximately the same size and large. The use case (for me at least) is to create several large sparse matrices which will be combined in various ways through Kronecker products. The combination happens at every time step in an evolution, so it really needs to be fast as well. I’m thinking mpi/petsc is probably not the most optimal way for dealing with this, and might just have to work with single node multi-threading.<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Best,<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Tyler<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;"><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Matthew Knepley <<a href="mailto:knepley@gmail.com" style="color: blue; text-decoration: underline;">knepley@gmail.com</a>><br><b>Date:<span class="Apple-converted-space"> </span></b>Monday, January 30, 2023 at 11:31 AM<br><b>To:<span class="Apple-converted-space"> </span></b>Guglielmo, Tyler Hardy <<a href="mailto:guglielmo2@llnl.gov" style="color: blue; text-decoration: underline;">guglielmo2@llnl.gov</a>><br><b>Cc:<span class="Apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>>,<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] Kronecker Product<o:p></o:p></span></p></div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Mon, Jan 30, 2023 at 2:24 PM Guglielmo, Tyler Hardy via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></span></div></div><div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;"><div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thanks Barry,<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I saw that function, but wasn’t sure how to apply it since the documentation says that S and T are dense matrices, but in my case all matrices involved are sparse. Is there a way to work around the dense requirement?<o:p></o:p></span></div></div></div></div></blockquote><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">We don't have parallel sparse-sparse. It would not be too hard to write, but it would be some work.<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">It is hard to understand the use case. Is one matrix much smaller? If not, and you inherit the distribution from A, it seems<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">like it might be very suboptimal, and otherwise you would have to redistribute on the fly and it would get very complicated.<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Thanks,<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Matt<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;"><div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Best,<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Tyler<o:p></o:p></span></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;"><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Date:<span class="Apple-converted-space"> </span></b>Monday, January 30, 2023 at 11:12 AM<br><b>To:<span class="Apple-converted-space"> </span></b>Guglielmo, Tyler Hardy <<a href="mailto:guglielmo2@llnl.gov" target="_blank" style="color: blue; text-decoration: underline;">guglielmo2@llnl.gov</a>><br><b>Cc:<span class="Apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] Kronecker Product</span><span style="font-size: 11pt;"><o:p></o:p></span></p></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Do you need the explicit sparse representation of the Kronecker product? Or do you want to apply it as an operator or solve systems with it? If the latter you can use <a href="https://urldefense.us/v3/__https:/petsc.org/release/docs/manualpages/Mat/MatCreateKAIJ/*matcreatekaij__;Iw!!G2kpM7uM-TzIFchu!lSQ9WFlYi6PMdfs3WAfEq4ydgCLZtfDgyFy9PjdLNTisCsHtwmVuukcpIv1J0i1EtiQ$" target="_blank" style="color: blue; text-decoration: underline;">https://petsc.org/release/docs/manualpages/Mat/MatCreateKAIJ/#matcreatekaij</a><o:p></o:p></span></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Barry<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div><div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></p><div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Jan 30, 2023, at 12:53 PM, Guglielmo, Tyler Hardy via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></span></div></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Hi all,<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I am wondering if there is any functionality for taking Kronecker products of large sparse matrices that are parallel? MatSeqAIJKron is as close as I have found, but it seems like this does not work for parallel matrices. Any ideas here? <o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">An option could be to make A and B sequential, compute the Kronecker product, C, then scatter C into a parallel matrix? This seems like a horribly inefficient procedure. I’m still fairly new to petsc, so thanks for patience :)!<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Best,<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Tyler<span class="m1068516836359424776apple-converted-space"> </span><o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div><div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">+++++++++++++++++++++++++++++<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Tyler Guglielmo<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Postdoctoral Researcher<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Lawrence Livermore National Lab<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Office: 925-423-6186<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Cell: 210-480-8000<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">+++++++++++++++++++++++++++++<o:p></o:p></span></div></div></div></div></div></blockquote></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> <o:p></o:p></span></div></div></div></div></div></div></blockquote></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br clear="all"><o:p></o:p></span></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">--<span class="Apple-converted-space"> </span><o:p></o:p></span></div><div><div><div><div><div><div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><a href="https://urldefense.us/v3/__http:/www.cse.buffalo.edu/*knepley/__;fg!!G2kpM7uM-TzIFchu!nK03nUENfNtHoOs8RWmJWJQYJH2IlC_lYQPNn7kV9FsBv2CQKR_VSqbLGRLFShVpKmY$" target="_blank" style="color: blue; text-decoration: underline;">https://www.cse.buffalo.edu/~knepley/</a></span></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></body></html>