<div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Dec 11, 2021 at 10:28 AM Rohan Yadav <<a href="mailto:rohany@alumni.cmu.edu">rohany@alumni.cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div><br></div><div>Thanks for the response!</div><div><br></div><div>> You can use <a href="https://petsc.org/main/src/mat/tests/ex72.c.html" target="_blank">https://petsc.org/main/src/mat/tests/ex72.c.html</a> to convert a <span style="font-family:"Helvetica Neue";font-size:12px">Matrix Market file into a petsc binary file. And then in your test, load the binary matrix, following this example </span><a href="https://petsc.org/main/src/mat/tutorials/ex1.c.html" target="_blank">https://petsc.org/main/src/mat/tutorials/ex1.c.html</a></div><div><br></div><div>I tried an example like this, but the performance was too slow (it would process ~2000-3000 calls to `SetValue` a second), which is not reasonable for loading matrices with millions of non-zeros.</div></div></blockquote><div>The matrix market file in text format is not good for load. One should convert it to petsc binary format (only once), and use the new binary file afterwards. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>> I don't know what "No Races" means, but it seems you'd better also verify the result of SpMV. </div><div><br></div><div>This is a correct implementation of SpMV. The no-races is fine as it parallelizes over the rows of the matrix, and thus does not need synchronization between writes to the output.</div><div><br></div><span style="color:rgb(80,0,80)"><div>> <span style="color:rgb(34,34,34)">You can think petsc's default CSR spmv is the baseline, which is done in ~10 lines of code. </span></div><div><span style="color:rgb(34,34,34)"><br></span></div><div><span style="color:rgb(34,34,34)">I'm sorry, but I don't think that is a reasonable statement w.r.t to the lines of code making it a good baseline. The TACO compiler also can be used in 10 lines of code to compute an SpMV, or any other state-of-the-art library could wrap an SpMV implementation behind a single function call. I'm wondering if this performance I'm seeing using PETSc is expected, or if I've misconfigured or am misusing the system in some way.</span></div></span></div></blockquote><div>I meant 10 lines of code without any function call, which can be thought of as a textbook implementation of SpMV. As a baseline, one can apply optimizations to it. PETSc does not do sophisticated sparse matrix optimization itself, instead it relies on third-party libraries. I remember we had OSKI from Berkeley for CPU, and on GPU we use cuSparse, hipSparse, MKLSparse or Kokkos-Kernels. If TACO is good, then petsc can add an interface to it too.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span style="color:rgb(80,0,80)"><div><span style="color:rgb(34,34,34)"></span></div><div><span style="color:rgb(34,34,34)">Rohan</span></div></span><div style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 10, 2021 at 11:39 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Dec 10, 2021 at 8:05 PM Rohan Yadav <<a href="mailto:rohany@alumni.cmu.edu" target="_blank">rohany@alumni.cmu.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">Hi, I’m Rohan, a student working on compilation techniques for distributed tensor computations. I’m looking at using PETSc as a baseline for experiments I’m running, and want to understand if I’m using PETSc as it was intended to achieve high performance, and if the performance I’m seeing is expected. Currently, I’m just looking at SpMV operations.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">My experiments are run on the Lassen Supercomputer (<a href="https://hpc.llnl.gov/hardware/platforms/lassen" target="_blank"><span style="color:rgb(220,161,13)">https://hpc.llnl.gov/hardware/platforms/lassen</span></a>). The system has 40 CPUs, 4 V100s and an Infiniband interconnect. A visualization of the architecture is here: <a href="https://hpc.llnl.gov/sites/default/files/power9-AC922systemDiagram2_1.png" target="_blank"><span style="color:rgb(220,161,13)" title="">https://hpc.llnl.gov/sites/default/files/power9-AC922systemDiagram2_1.png</span></a>.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">As of now, I’m trying to understand the single-node performance of PETSc, as the scaling performance onto multiple nodes appears to be as I expect. I’m using the arabic-2005 sparse matrix from the SuiteSparse matrix collection, detailed here: <a href="https://sparse.tamu.edu/LAW/arabic-2005" target="_blank"><span style="color:rgb(220,161,13)">https://sparse.tamu.edu/LAW/arabic-2005</span></a>. As a trusted baseline, I am comparing against SpMV code generated by the TACO compiler (<a href="http://tensor-compiler.org/codegen.html?expr=y(i)%20=%20A(i,j)%20*%20x(j)&format=y:d:0;A:ds:0,1;x:d:0&sched=split:i:i0:i1:32;reorder:i0:i1:j;parallelize:i0:CPU%20Thread:No%20Races)" target="_blank"><span style="color:rgb(220,161,13)">http://tensor-compiler.org/codegen.html?expr=y(i)%20=%20A(i,j)%20*%20x(j)&format=y:d:0;A:ds:0,1;x:d:0&sched=split:i:i0:i1:32;reorder:i0:i1:j;parallelize:i0:CPU%20Thread:No%20Races)</span></a>.</p></div></blockquote><div>I don't know what "No Races" means, but it seems you'd better also verify the result of SpMV. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">My experiments find that PETSc is roughly 4 times slower on a single thread and node than the kernel generated by TACO:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">PETSc: 1 Thread: 5694.72 ms, 1 Node 40 threads: 262.6 ms.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">TACO: 1 Thread: 1341 ms, 1 Node 40 threads: 86 ms.</p></div></blockquote><div>You can think petsc's default CSR spmv is the baseline, which is done in ~10 lines of code. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(220,161,13)"><span style="color:rgb(0,0,0)">My code using PETSc is here: <a href="https://github.com/rohany/taco/blob/9e0e30b16bfba5319b15b2d1392f35376952f838/petsc/benchmark.cpp#L38" target="_blank">https://github.com/rohany/taco/blob/9e0e30b16bfba5319b15b2d1392f35376952f838/petsc/benchmark.cpp#L38</a></span>.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(220,161,13);min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">Runs from 1 thread and 1 node with -log_view are attached to the email. The command lines for each were as follows:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">1 node 1 thread: `jsrun -n 1 -c 1 -r 1 -b rs ./bin/benchmark -n 20 -warmup 10 -matrix $TENSOR_DIR/arabic-2005.petsc -log_view`</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">1 node 40 threads: `jsrun -n 40 -c 1 -r 40 -b rs ./bin/benchmark -n 20 -warmup 10 -matrix $TENSOR_DIR/arabic-2005.petsc -log_view`</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">In addition to these benchmarking concerns, I wanted to share my experiences trying to load data from Matrix Market files into PETSc, which ended up 1being much more difficult than I anticipated. Essentially, trying to iterate through the Matrix Market files and using `write` to insert entries into a `Mat` was extremely slow. In order to get reasonable performance, I had to use an external utility to basically construct a CSR matrix, and then pass the arrays from the CSR Matrix into `MatCreateSeqAIJWithArrays`. I couldn’t find any more guidance on PETSc forums or Google, so I wanted to know if this was the right way to go.</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">Thanks,</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">Rohan Yadav</p></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>