<div dir="ltr">Hi, Chang,<div> I revised your patch and made an MR <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4471">https://gitlab.com/petsc/petsc/-/merge_requests/4471</a> with branch jczhang/fix-PCTelescope-GPU </div><div> Could you check if it works on your end?</div><div> Thanks.</div><div> --Junchao Zhang<br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 16, 2021 at 8:59 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div>Hi, Chang,</div><div> Thanks a lot for the fix. I will create an MR for it.</div><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 16, 2021 at 8:12 PM Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Barry, Pierre and Junchao,<br>
<br>
I spent some time to find the reason for the error. I think it is caused <br>
by some compability issues between telescope and cusparse.<br>
<br>
1. In PCTelescopeMatCreate_default in telescope.c, it calls <br>
MatCreateMPIMatConcatenateSeqMat to concat seqmat to mpimat, but this <br>
function is from mpiaij.c and will set the mat type to mpiaij, even if <br>
the original matrix is mpiaijcusparse.<br>
<br>
2. Simiar issue exists in PCTelescopeSetUp_default, where the vector is <br>
set to type mpi rather than mpicuda.<br>
<br>
I have fixed the issue using the following patch. After applying it, <br>
telescope and cusparse work as expected.<br>
<br>
diff --git a/src/ksp/pc/impls/telescope/telescope.c <br>
b/src/ksp/pc/impls/telescope/telescope.c<br>
index 893febb055..d3f687eff9 100644<br>
--- a/src/ksp/pc/impls/telescope/telescope.c<br>
+++ b/src/ksp/pc/impls/telescope/telescope.c<br>
@@ -159,6 +159,7 @@ PetscErrorCode PCTelescopeSetUp_default(PC <br>
pc,PC_Telescope sred)<br>
ierr = VecCreate(subcomm,&xred);CHKERRQ(ierr);<br>
ierr = VecSetSizes(xred,PETSC_DECIDE,M);CHKERRQ(ierr);<br>
ierr = VecSetBlockSize(xred,bs);CHKERRQ(ierr);<br>
+ ierr = VecSetType(xred,((PetscObject)x)->type_name);CHKERRQ(ierr);<br>
ierr = VecSetFromOptions(xred);CHKERRQ(ierr);<br>
ierr = VecGetLocalSize(xred,&m);CHKERRQ(ierr);<br>
}<br>
diff --git a/src/mat/impls/aij/mpi/mpiaij.c b/src/mat/impls/aij/mpi/mpiaij.c<br>
index 36077002db..ac374e07eb 100644<br>
--- a/src/mat/impls/aij/mpi/mpiaij.c<br>
+++ b/src/mat/impls/aij/mpi/mpiaij.c<br>
@@ -4486,6 +4486,7 @@ PetscErrorCode <br>
MatCreateMPIMatConcatenateSeqMat_MPIAIJ(MPI_Comm comm,Mat inmat,P<br>
PetscInt m,N,i,rstart,nnz,Ii;<br>
PetscInt *indx;<br>
PetscScalar *values;<br>
+ PetscBool isseqaijcusparse;<br>
<br>
PetscFunctionBegin;<br>
ierr = MatGetSize(inmat,&m,&N);CHKERRQ(ierr);<br>
@@ -4513,7 +4514,12 @@ PetscErrorCode <br>
MatCreateMPIMatConcatenateSeqMat_MPIAIJ(MPI_Comm comm,Mat inmat,P<br>
ierr = <br>
MatSetSizes(*outmat,m,n,PETSC_DETERMINE,PETSC_DETERMINE);CHKERRQ(ierr);<br>
ierr = MatGetBlockSizes(inmat,&bs,&cbs);CHKERRQ(ierr);<br>
ierr = MatSetBlockSizes(*outmat,bs,cbs);CHKERRQ(ierr);<br>
- ierr = MatSetType(*outmat,MATAIJ);CHKERRQ(ierr);<br>
+ ierr = <br>
PetscObjectBaseTypeCompare((PetscObject)inmat,MATSEQAIJCUSPARSE,&isseqaijcusparse);CHKERRQ(ierr);<br>
+ if (isseqaijcusparse) {<br>
+ ierr = MatSetType(*outmat,MATAIJCUSPARSE);CHKERRQ(ierr);<br>
+ } else {<br>
+ ierr = MatSetType(*outmat,MATAIJ);CHKERRQ(ierr);<br>
+ }<br>
ierr = MatSeqAIJSetPreallocation(*outmat,0,dnz);CHKERRQ(ierr);<br>
ierr = MatMPIAIJSetPreallocation(*outmat,0,dnz,0,onz);CHKERRQ(ierr);<br>
ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);<br>
<br>
Please help view it and merge to master if possible.<br>
<br>
Regards,<br>
<br>
Chang<br>
<br>
On 10/15/21 1:27 PM, Barry Smith wrote:<br>
> <br>
> So the only difference is between <br>
> -sub_telescope_pc_factor_mat_solver_type cusparse and <br>
> -sub_telescope_pc_factor_mat_solver_type mumps ?<br>
> <br>
> Try without the -sub_telescope_pc_factor_mat_solver_type cusparse <br>
> and then PETSc will just use the CPU solvers, I want to see if that <br>
> works, it should. If it works then there is perhaps something specific <br>
> about the PCTELESCOPE and the cusparse solver, for example the right <br>
> hand side array values may never get to the GPU.<br>
> <br>
> Barry<br>
> <br>
>> On Oct 14, 2021, at 10:11 PM, Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> wrote:<br>
>><br>
>> For comparison, here is the output using mumps instead of cusparse<br>
>><br>
>> $ mpiexec -n 16 --hostfile hostfile --oversubscribe ./ex7 -m 400 <br>
>> -ksp_view -ksp_monitor_true_residual -pc_type bjacobi <br>
>> -pc_bjacobi_blocks 4 -ksp_type fgmres -mat_type aijcusparse <br>
>> -sub_pc_type telescope -sub_ksp_type preonly -sub_telescope_ksp_type <br>
>> preonly -sub_telescope_pc_type lu <br>
>> -sub_telescope_pc_factor_mat_solver_type mumps <br>
>> -sub_pc_telescope_reduction_factor 4 -sub_pc_telescope_subcomm_type <br>
>> contiguous -ksp_max_it 2000 -ksp_rtol 1.e-20 -ksp_atol 1.e-9<br>
> <br>
> $ mpiexec -n 16 --hostfile hostfile --oversubscribe ./ex7 -m 400 <br>
> -ksp_view -ksp_monitor_true_residual -pc_type bjacobi -pc_bjacobi_blocks <br>
> 4 -ksp_type fgmres -mat_type aijcusparse -sub_pc_type telescope <br>
> -sub_ksp_type preonly -sub_telescope_ksp_type preonly <br>
> -sub_telescope_pc_type lu -sub_telescope_pc_factor_mat_solver_type <br>
> cusparse -sub_pc_telescope_reduction_factor 4 <br>
> -sub_pc_telescope_subcomm_type contiguous -ksp_max_it 2000 -ksp_rtol <br>
> 1.e-20 -ksp_atol 1.e-9<br>
> <br>
> <br>
>> 0 KSP unpreconditioned resid norm 4.014971979977e+01 true resid norm <br>
>> 4.014971979977e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
>> 1 KSP unpreconditioned resid norm 2.439995191694e+00 true resid norm <br>
>> 2.439995191694e+00 ||r(i)||/||b|| 6.077240896978e-02<br>
>> 2 KSP unpreconditioned resid norm 1.280694102588e+00 true resid norm <br>
>> 1.280694102588e+00 ||r(i)||/||b|| 3.189795866509e-02<br>
>> 3 KSP unpreconditioned resid norm 1.041100266810e+00 true resid norm <br>
>> 1.041100266810e+00 ||r(i)||/||b|| 2.593044912896e-02<br>
>> 4 KSP unpreconditioned resid norm 7.274347137268e-01 true resid norm <br>
>> 7.274347137268e-01 ||r(i)||/||b|| 1.811805206499e-02<br>
>> 5 KSP unpreconditioned resid norm 5.429229329787e-01 true resid norm <br>
>> 5.429229329787e-01 ||r(i)||/||b|| 1.352245882876e-02<br>
>> 6 KSP unpreconditioned resid norm 4.332970410353e-01 true resid norm <br>
>> 4.332970410353e-01 ||r(i)||/||b|| 1.079203150598e-02<br>
>> 7 KSP unpreconditioned resid norm 3.948206050950e-01 true resid norm <br>
>> 3.948206050950e-01 ||r(i)||/||b|| 9.833707609019e-03<br>
>> 8 KSP unpreconditioned resid norm 3.379580577269e-01 true resid norm <br>
>> 3.379580577269e-01 ||r(i)||/||b|| 8.417444988714e-03<br>
>> 9 KSP unpreconditioned resid norm 2.875593971410e-01 true resid norm <br>
>> 2.875593971410e-01 ||r(i)||/||b|| 7.162176936105e-03<br>
>> 10 KSP unpreconditioned resid norm 2.533983363244e-01 true resid norm <br>
>> 2.533983363244e-01 ||r(i)||/||b|| 6.311335112378e-03<br>
>> 11 KSP unpreconditioned resid norm 2.389169921094e-01 true resid norm <br>
>> 2.389169921094e-01 ||r(i)||/||b|| 5.950651543793e-03<br>
>> 12 KSP unpreconditioned resid norm 2.118961639089e-01 true resid norm <br>
>> 2.118961639089e-01 ||r(i)||/||b|| 5.277649880637e-03<br>
>> 13 KSP unpreconditioned resid norm 1.885892030223e-01 true resid norm <br>
>> 1.885892030223e-01 ||r(i)||/||b|| 4.697148671593e-03<br>
>> 14 KSP unpreconditioned resid norm 1.763510666948e-01 true resid norm <br>
>> 1.763510666948e-01 ||r(i)||/||b|| 4.392336175055e-03<br>
>> 15 KSP unpreconditioned resid norm 1.638219366731e-01 true resid norm <br>
>> 1.638219366731e-01 ||r(i)||/||b|| 4.080275964317e-03<br>
>> 16 KSP unpreconditioned resid norm 1.476792766432e-01 true resid norm <br>
>> 1.476792766432e-01 ||r(i)||/||b|| 3.678214378076e-03<br>
>> 17 KSP unpreconditioned resid norm 1.349906937321e-01 true resid norm <br>
>> 1.349906937321e-01 ||r(i)||/||b|| 3.362182710248e-03<br>
>> 18 KSP unpreconditioned resid norm 1.289673236836e-01 true resid norm <br>
>> 1.289673236836e-01 ||r(i)||/||b|| 3.212159993314e-03<br>
>> 19 KSP unpreconditioned resid norm 1.167505658153e-01 true resid norm <br>
>> 1.167505658153e-01 ||r(i)||/||b|| 2.907879965230e-03<br>
>> 20 KSP unpreconditioned resid norm 1.046037988999e-01 true resid norm <br>
>> 1.046037988999e-01 ||r(i)||/||b|| 2.605343185995e-03<br>
>> 21 KSP unpreconditioned resid norm 9.832660514331e-02 true resid norm <br>
>> 9.832660514331e-02 ||r(i)||/||b|| 2.448998539309e-03<br>
>> 22 KSP unpreconditioned resid norm 8.835618950141e-02 true resid norm <br>
>> 8.835618950142e-02 ||r(i)||/||b|| 2.200667649539e-03<br>
>> 23 KSP unpreconditioned resid norm 7.563496650115e-02 true resid norm <br>
>> 7.563496650116e-02 ||r(i)||/||b|| 1.883823022386e-03<br>
>> 24 KSP unpreconditioned resid norm 6.651291376834e-02 true resid norm <br>
>> 6.651291376834e-02 ||r(i)||/||b|| 1.656622115921e-03<br>
>> 25 KSP unpreconditioned resid norm 5.890393227906e-02 true resid norm <br>
>> 5.890393227906e-02 ||r(i)||/||b|| 1.467106933070e-03<br>
>> 26 KSP unpreconditioned resid norm 4.661992782780e-02 true resid norm <br>
>> 4.661992782780e-02 ||r(i)||/||b|| 1.161152009536e-03<br>
>> 27 KSP unpreconditioned resid norm 3.690705358716e-02 true resid norm <br>
>> 3.690705358716e-02 ||r(i)||/||b|| 9.192356452602e-04<br>
>> 28 KSP unpreconditioned resid norm 3.209680460188e-02 true resid norm <br>
>> 3.209680460188e-02 ||r(i)||/||b|| 7.994278605666e-04<br>
>> 29 KSP unpreconditioned resid norm 2.354337626000e-02 true resid norm <br>
>> 2.354337626001e-02 ||r(i)||/||b|| 5.863895533373e-04<br>
>> 30 KSP unpreconditioned resid norm 1.701296561785e-02 true resid norm <br>
>> 1.701296561785e-02 ||r(i)||/||b|| 4.237380908932e-04<br>
>> 31 KSP unpreconditioned resid norm 1.509942937258e-02 true resid norm <br>
>> 1.509942937258e-02 ||r(i)||/||b|| 3.760780759588e-04<br>
>> 32 KSP unpreconditioned resid norm 1.258274688515e-02 true resid norm <br>
>> 1.258274688515e-02 ||r(i)||/||b|| 3.133956338402e-04<br>
>> 33 KSP unpreconditioned resid norm 9.805748771638e-03 true resid norm <br>
>> 9.805748771638e-03 ||r(i)||/||b|| 2.442295692359e-04<br>
>> 34 KSP unpreconditioned resid norm 8.596552678160e-03 true resid norm <br>
>> 8.596552678160e-03 ||r(i)||/||b|| 2.141123953301e-04<br>
>> 35 KSP unpreconditioned resid norm 6.936406707500e-03 true resid norm <br>
>> 6.936406707500e-03 ||r(i)||/||b|| 1.727635147167e-04<br>
>> 36 KSP unpreconditioned resid norm 5.533741607932e-03 true resid norm <br>
>> 5.533741607932e-03 ||r(i)||/||b|| 1.378276519869e-04<br>
>> 37 KSP unpreconditioned resid norm 4.982347757923e-03 true resid norm <br>
>> 4.982347757923e-03 ||r(i)||/||b|| 1.240942099414e-04<br>
>> 38 KSP unpreconditioned resid norm 4.309608348059e-03 true resid norm <br>
>> 4.309608348059e-03 ||r(i)||/||b|| 1.073384414524e-04<br>
>> 39 KSP unpreconditioned resid norm 3.729408303186e-03 true resid norm <br>
>> 3.729408303185e-03 ||r(i)||/||b|| 9.288753001974e-05<br>
>> 40 KSP unpreconditioned resid norm 3.490003351128e-03 true resid norm <br>
>> 3.490003351128e-03 ||r(i)||/||b|| 8.692472496776e-05<br>
>> 41 KSP unpreconditioned resid norm 3.069012426454e-03 true resid norm <br>
>> 3.069012426453e-03 ||r(i)||/||b|| 7.643919912166e-05<br>
>> 42 KSP unpreconditioned resid norm 2.772928845284e-03 true resid norm <br>
>> 2.772928845284e-03 ||r(i)||/||b|| 6.906471225983e-05<br>
>> 43 KSP unpreconditioned resid norm 2.561454192399e-03 true resid norm <br>
>> 2.561454192398e-03 ||r(i)||/||b|| 6.379756085902e-05<br>
>> 44 KSP unpreconditioned resid norm 2.253662762802e-03 true resid norm <br>
>> 2.253662762802e-03 ||r(i)||/||b|| 5.613146926159e-05<br>
>> 45 KSP unpreconditioned resid norm 2.086800523919e-03 true resid norm <br>
>> 2.086800523919e-03 ||r(i)||/||b|| 5.197546917701e-05<br>
>> 46 KSP unpreconditioned resid norm 1.926028182896e-03 true resid norm <br>
>> 1.926028182896e-03 ||r(i)||/||b|| 4.797114880257e-05<br>
>> 47 KSP unpreconditioned resid norm 1.769243808622e-03 true resid norm <br>
>> 1.769243808622e-03 ||r(i)||/||b|| 4.406615581492e-05<br>
>> 48 KSP unpreconditioned resid norm 1.656654905964e-03 true resid norm <br>
>> 1.656654905964e-03 ||r(i)||/||b|| 4.126192945371e-05<br>
>> 49 KSP unpreconditioned resid norm 1.572052627273e-03 true resid norm <br>
>> 1.572052627273e-03 ||r(i)||/||b|| 3.915475961260e-05<br>
>> 50 KSP unpreconditioned resid norm 1.454960682355e-03 true resid norm <br>
>> 1.454960682355e-03 ||r(i)||/||b|| 3.623837699518e-05<br>
>> 51 KSP unpreconditioned resid norm 1.375985053014e-03 true resid norm <br>
>> 1.375985053014e-03 ||r(i)||/||b|| 3.427134883820e-05<br>
>> 52 KSP unpreconditioned resid norm 1.269325501087e-03 true resid norm <br>
>> 1.269325501087e-03 ||r(i)||/||b|| 3.161480347603e-05<br>
>> 53 KSP unpreconditioned resid norm 1.184791772965e-03 true resid norm <br>
>> 1.184791772965e-03 ||r(i)||/||b|| 2.950934100844e-05<br>
>> 54 KSP unpreconditioned resid norm 1.064535156080e-03 true resid norm <br>
>> 1.064535156080e-03 ||r(i)||/||b|| 2.651413662135e-05<br>
>> 55 KSP unpreconditioned resid norm 9.639036688120e-04 true resid norm <br>
>> 9.639036688117e-04 ||r(i)||/||b|| 2.400773090370e-05<br>
>> 56 KSP unpreconditioned resid norm 8.632359780260e-04 true resid norm <br>
>> 8.632359780260e-04 ||r(i)||/||b|| 2.150042347322e-05<br>
>> 57 KSP unpreconditioned resid norm 7.613605783850e-04 true resid norm <br>
>> 7.613605783850e-04 ||r(i)||/||b|| 1.896303591113e-05<br>
>> 58 KSP unpreconditioned resid norm 6.681073248348e-04 true resid norm <br>
>> 6.681073248349e-04 ||r(i)||/||b|| 1.664039819373e-05<br>
>> 59 KSP unpreconditioned resid norm 5.656127908544e-04 true resid norm <br>
>> 5.656127908545e-04 ||r(i)||/||b|| 1.408758999254e-05<br>
>> 60 KSP unpreconditioned resid norm 4.850863370767e-04 true resid norm <br>
>> 4.850863370767e-04 ||r(i)||/||b|| 1.208193580169e-05<br>
>> 61 KSP unpreconditioned resid norm 4.374055762320e-04 true resid norm <br>
>> 4.374055762316e-04 ||r(i)||/||b|| 1.089436186387e-05<br>
>> 62 KSP unpreconditioned resid norm 3.874398257079e-04 true resid norm <br>
>> 3.874398257077e-04 ||r(i)||/||b|| 9.649876204364e-06<br>
>> 63 KSP unpreconditioned resid norm 3.364908694427e-04 true resid norm <br>
>> 3.364908694429e-04 ||r(i)||/||b|| 8.380902061609e-06<br>
>> 64 KSP unpreconditioned resid norm 2.961034697265e-04 true resid norm <br>
>> 2.961034697268e-04 ||r(i)||/||b|| 7.374982221632e-06<br>
>> 65 KSP unpreconditioned resid norm 2.640593092764e-04 true resid norm <br>
>> 2.640593092767e-04 ||r(i)||/||b|| 6.576865557059e-06<br>
>> 66 KSP unpreconditioned resid norm 2.423231125743e-04 true resid norm <br>
>> 2.423231125745e-04 ||r(i)||/||b|| 6.035487016671e-06<br>
>> 67 KSP unpreconditioned resid norm 2.182349471179e-04 true resid norm <br>
>> 2.182349471179e-04 ||r(i)||/||b|| 5.435528521898e-06<br>
>> 68 KSP unpreconditioned resid norm 2.008438265031e-04 true resid norm <br>
>> 2.008438265028e-04 ||r(i)||/||b|| 5.002371809927e-06<br>
>> 69 KSP unpreconditioned resid norm 1.838732863386e-04 true resid norm <br>
>> 1.838732863388e-04 ||r(i)||/||b|| 4.579690400226e-06<br>
>> 70 KSP unpreconditioned resid norm 1.723786027645e-04 true resid norm <br>
>> 1.723786027645e-04 ||r(i)||/||b|| 4.293394913444e-06<br>
>> 71 KSP unpreconditioned resid norm 1.580945192204e-04 true resid norm <br>
>> 1.580945192205e-04 ||r(i)||/||b|| 3.937624471826e-06<br>
>> 72 KSP unpreconditioned resid norm 1.476687469671e-04 true resid norm <br>
>> 1.476687469671e-04 ||r(i)||/||b|| 3.677952117812e-06<br>
>> 73 KSP unpreconditioned resid norm 1.385018526182e-04 true resid norm <br>
>> 1.385018526184e-04 ||r(i)||/||b|| 3.449634351350e-06<br>
>> 74 KSP unpreconditioned resid norm 1.279712893541e-04 true resid norm <br>
>> 1.279712893541e-04 ||r(i)||/||b|| 3.187351991305e-06<br>
>> 75 KSP unpreconditioned resid norm 1.202010411772e-04 true resid norm <br>
>> 1.202010411774e-04 ||r(i)||/||b|| 2.993820175504e-06<br>
>> 76 KSP unpreconditioned resid norm 1.113459414198e-04 true resid norm <br>
>> 1.113459414200e-04 ||r(i)||/||b|| 2.773268206485e-06<br>
>> 77 KSP unpreconditioned resid norm 1.042523036036e-04 true resid norm <br>
>> 1.042523036037e-04 ||r(i)||/||b|| 2.596588572066e-06<br>
>> 78 KSP unpreconditioned resid norm 9.565176453232e-05 true resid norm <br>
>> 9.565176453227e-05 ||r(i)||/||b|| 2.382376888539e-06<br>
>> 79 KSP unpreconditioned resid norm 8.896901670359e-05 true resid norm <br>
>> 8.896901670365e-05 ||r(i)||/||b|| 2.215931198209e-06<br>
>> 80 KSP unpreconditioned resid norm 8.119298425803e-05 true resid norm <br>
>> 8.119298425824e-05 ||r(i)||/||b|| 2.022255314935e-06<br>
>> 81 KSP unpreconditioned resid norm 7.544528309154e-05 true resid norm <br>
>> 7.544528309154e-05 ||r(i)||/||b|| 1.879098620558e-06<br>
>> 82 KSP unpreconditioned resid norm 6.755385041138e-05 true resid norm <br>
>> 6.755385041176e-05 ||r(i)||/||b|| 1.682548489719e-06<br>
>> 83 KSP unpreconditioned resid norm 6.158629300870e-05 true resid norm <br>
>> 6.158629300835e-05 ||r(i)||/||b|| 1.533915885727e-06<br>
>> 84 KSP unpreconditioned resid norm 5.358756885754e-05 true resid norm <br>
>> 5.358756885765e-05 ||r(i)||/||b|| 1.334693470462e-06<br>
>> 85 KSP unpreconditioned resid norm 4.774852370380e-05 true resid norm <br>
>> 4.774852370387e-05 ||r(i)||/||b|| 1.189261692037e-06<br>
>> 86 KSP unpreconditioned resid norm 3.919358737908e-05 true resid norm <br>
>> 3.919358737930e-05 ||r(i)||/||b|| 9.761858258229e-07<br>
>> 87 KSP unpreconditioned resid norm 3.434042319950e-05 true resid norm <br>
>> 3.434042319947e-05 ||r(i)||/||b|| 8.553091620745e-07<br>
>> 88 KSP unpreconditioned resid norm 2.813699436281e-05 true resid norm <br>
>> 2.813699436302e-05 ||r(i)||/||b|| 7.008017615898e-07<br>
>> 89 KSP unpreconditioned resid norm 2.462248069068e-05 true resid norm <br>
>> 2.462248069051e-05 ||r(i)||/||b|| 6.132665635851e-07<br>
>> 90 KSP unpreconditioned resid norm 2.040558789626e-05 true resid norm <br>
>> 2.040558789626e-05 ||r(i)||/||b|| 5.082373674841e-07<br>
>> 91 KSP unpreconditioned resid norm 1.888523204468e-05 true resid norm <br>
>> 1.888523204470e-05 ||r(i)||/||b|| 4.703702077842e-07<br>
>> 92 KSP unpreconditioned resid norm 1.707071292484e-05 true resid norm <br>
>> 1.707071292474e-05 ||r(i)||/||b|| 4.251763900191e-07<br>
>> 93 KSP unpreconditioned resid norm 1.498636454665e-05 true resid norm <br>
>> 1.498636454672e-05 ||r(i)||/||b|| 3.732619958859e-07<br>
>> 94 KSP unpreconditioned resid norm 1.219393542993e-05 true resid norm <br>
>> 1.219393543006e-05 ||r(i)||/||b|| 3.037115947725e-07<br>
>> 95 KSP unpreconditioned resid norm 1.059996963300e-05 true resid norm <br>
>> 1.059996963303e-05 ||r(i)||/||b|| 2.640110487917e-07<br>
>> 96 KSP unpreconditioned resid norm 9.099659872548e-06 true resid norm <br>
>> 9.099659873214e-06 ||r(i)||/||b|| 2.266431725699e-07<br>
>> 97 KSP unpreconditioned resid norm 8.147347587295e-06 true resid norm <br>
>> 8.147347587584e-06 ||r(i)||/||b|| 2.029241456283e-07<br>
>> 98 KSP unpreconditioned resid norm 7.167226146744e-06 true resid norm <br>
>> 7.167226146783e-06 ||r(i)||/||b|| 1.785124823418e-07<br>
>> 99 KSP unpreconditioned resid norm 6.552540209538e-06 true resid norm <br>
>> 6.552540209577e-06 ||r(i)||/||b|| 1.632026385802e-07<br>
>> 100 KSP unpreconditioned resid norm 5.767783600111e-06 true resid norm <br>
>> 5.767783600320e-06 ||r(i)||/||b|| 1.436568830140e-07<br>
>> 101 KSP unpreconditioned resid norm 5.261057430584e-06 true resid norm <br>
>> 5.261057431144e-06 ||r(i)||/||b|| 1.310359688033e-07<br>
>> 102 KSP unpreconditioned resid norm 4.715498525786e-06 true resid norm <br>
>> 4.715498525947e-06 ||r(i)||/||b|| 1.174478564100e-07<br>
>> 103 KSP unpreconditioned resid norm 4.380052669622e-06 true resid norm <br>
>> 4.380052669825e-06 ||r(i)||/||b|| 1.090929822591e-07<br>
>> 104 KSP unpreconditioned resid norm 3.911664470060e-06 true resid norm <br>
>> 3.911664470226e-06 ||r(i)||/||b|| 9.742694319496e-08<br>
>> 105 KSP unpreconditioned resid norm 3.652211458315e-06 true resid norm <br>
>> 3.652211458259e-06 ||r(i)||/||b|| 9.096480564430e-08<br>
>> 106 KSP unpreconditioned resid norm 3.387532128049e-06 true resid norm <br>
>> 3.387532128358e-06 ||r(i)||/||b|| 8.437249737363e-08<br>
>> 107 KSP unpreconditioned resid norm 3.234218880987e-06 true resid norm <br>
>> 3.234218880798e-06 ||r(i)||/||b|| 8.055395895481e-08<br>
>> 108 KSP unpreconditioned resid norm 3.016905196388e-06 true resid norm <br>
>> 3.016905196492e-06 ||r(i)||/||b|| 7.514137611763e-08<br>
>> 109 KSP unpreconditioned resid norm 2.858246441921e-06 true resid norm <br>
>> 2.858246441975e-06 ||r(i)||/||b|| 7.118969836476e-08<br>
>> 110 KSP unpreconditioned resid norm 2.637118810847e-06 true resid norm <br>
>> 2.637118810750e-06 ||r(i)||/||b|| 6.568212241336e-08<br>
>> 111 KSP unpreconditioned resid norm 2.494976088717e-06 true resid norm <br>
>> 2.494976088700e-06 ||r(i)||/||b|| 6.214180574966e-08<br>
>> 112 KSP unpreconditioned resid norm 2.270639574272e-06 true resid norm <br>
>> 2.270639574200e-06 ||r(i)||/||b|| 5.655430686750e-08<br>
>> 113 KSP unpreconditioned resid norm 2.104988663813e-06 true resid norm <br>
>> 2.104988664169e-06 ||r(i)||/||b|| 5.242847707696e-08<br>
>> 114 KSP unpreconditioned resid norm 1.889361127301e-06 true resid norm <br>
>> 1.889361127526e-06 ||r(i)||/||b|| 4.705789073868e-08<br>
>> 115 KSP unpreconditioned resid norm 1.732367008052e-06 true resid norm <br>
>> 1.732367007971e-06 ||r(i)||/||b|| 4.314767367271e-08<br>
>> 116 KSP unpreconditioned resid norm 1.509288268391e-06 true resid norm <br>
>> 1.509288268645e-06 ||r(i)||/||b|| 3.759150191264e-08<br>
>> 117 KSP unpreconditioned resid norm 1.359169217644e-06 true resid norm <br>
>> 1.359169217445e-06 ||r(i)||/||b|| 3.385252062089e-08<br>
>> 118 KSP unpreconditioned resid norm 1.180146337735e-06 true resid norm <br>
>> 1.180146337908e-06 ||r(i)||/||b|| 2.939363820703e-08<br>
>> 119 KSP unpreconditioned resid norm 1.067757039683e-06 true resid norm <br>
>> 1.067757039924e-06 ||r(i)||/||b|| 2.659438335433e-08<br>
>> 120 KSP unpreconditioned resid norm 9.435833073736e-07 true resid norm <br>
>> 9.435833073736e-07 ||r(i)||/||b|| 2.350161625235e-08<br>
>> 121 KSP unpreconditioned resid norm 8.749457237613e-07 true resid norm <br>
>> 8.749457236791e-07 ||r(i)||/||b|| 2.179207546261e-08<br>
>> 122 KSP unpreconditioned resid norm 7.945760150897e-07 true resid norm <br>
>> 7.945760150444e-07 ||r(i)||/||b|| 1.979032528762e-08<br>
>> 123 KSP unpreconditioned resid norm 7.141240839013e-07 true resid norm <br>
>> 7.141240838682e-07 ||r(i)||/||b|| 1.778652721438e-08<br>
>> 124 KSP unpreconditioned resid norm 6.300566936733e-07 true resid norm <br>
>> 6.300566936607e-07 ||r(i)||/||b|| 1.569267971988e-08<br>
>> 125 KSP unpreconditioned resid norm 5.628986997544e-07 true resid norm <br>
>> 5.628986995849e-07 ||r(i)||/||b|| 1.401999073448e-08<br>
>> 126 KSP unpreconditioned resid norm 5.119018951602e-07 true resid norm <br>
>> 5.119018951837e-07 ||r(i)||/||b|| 1.274982484900e-08<br>
>> 127 KSP unpreconditioned resid norm 4.664670343748e-07 true resid norm <br>
>> 4.664670344042e-07 ||r(i)||/||b|| 1.161818903670e-08<br>
>> 128 KSP unpreconditioned resid norm 4.253264691112e-07 true resid norm <br>
>> 4.253264691948e-07 ||r(i)||/||b|| 1.059351027394e-08<br>
>> 129 KSP unpreconditioned resid norm 3.868921150516e-07 true resid norm <br>
>> 3.868921150517e-07 ||r(i)||/||b|| 9.636234498800e-09<br>
>> 130 KSP unpreconditioned resid norm 3.558445658540e-07 true resid norm <br>
>> 3.558445660061e-07 ||r(i)||/||b|| 8.862940209315e-09<br>
>> 131 KSP unpreconditioned resid norm 3.268710273840e-07 true resid norm <br>
>> 3.268710272455e-07 ||r(i)||/||b|| 8.141302825416e-09<br>
>> 132 KSP unpreconditioned resid norm 3.041273897592e-07 true resid norm <br>
>> 3.041273896694e-07 ||r(i)||/||b|| 7.574832182794e-09<br>
>> 133 KSP unpreconditioned resid norm 2.851926677922e-07 true resid norm <br>
>> 2.851926674248e-07 ||r(i)||/||b|| 7.103229333782e-09<br>
>> 134 KSP unpreconditioned resid norm 2.694708315072e-07 true resid norm <br>
>> 2.694708309500e-07 ||r(i)||/||b|| 6.711649104748e-09<br>
>> 135 KSP unpreconditioned resid norm 2.534825559099e-07 true resid norm <br>
>> 2.534825557469e-07 ||r(i)||/||b|| 6.313432746507e-09<br>
>> 136 KSP unpreconditioned resid norm 2.387342352458e-07 true resid norm <br>
>> 2.387342351804e-07 ||r(i)||/||b|| 5.946099658254e-09<br>
>> 137 KSP unpreconditioned resid norm 2.200861667617e-07 true resid norm <br>
>> 2.200861665255e-07 ||r(i)||/||b|| 5.481636425438e-09<br>
>> 138 KSP unpreconditioned resid norm 2.051415370616e-07 true resid norm <br>
>> 2.051415370614e-07 ||r(i)||/||b|| 5.109413915824e-09<br>
>> 139 KSP unpreconditioned resid norm 1.887376429396e-07 true resid norm <br>
>> 1.887376426682e-07 ||r(i)||/||b|| 4.700845824315e-09<br>
>> 140 KSP unpreconditioned resid norm 1.729743133005e-07 true resid norm <br>
>> 1.729743128342e-07 ||r(i)||/||b|| 4.308232129561e-09<br>
>> 141 KSP unpreconditioned resid norm 1.541021130781e-07 true resid norm <br>
>> 1.541021128364e-07 ||r(i)||/||b|| 3.838186508023e-09<br>
>> 142 KSP unpreconditioned resid norm 1.384631628565e-07 true resid norm <br>
>> 1.384631627735e-07 ||r(i)||/||b|| 3.448670712125e-09<br>
>> 143 KSP unpreconditioned resid norm 1.223114405626e-07 true resid norm <br>
>> 1.223114403883e-07 ||r(i)||/||b|| 3.046383411846e-09<br>
>> 144 KSP unpreconditioned resid norm 1.087313066223e-07 true resid norm <br>
>> 1.087313065117e-07 ||r(i)||/||b|| 2.708146085550e-09<br>
>> 145 KSP unpreconditioned resid norm 9.181901998734e-08 true resid norm <br>
>> 9.181901984268e-08 ||r(i)||/||b|| 2.286915582489e-09<br>
>> 146 KSP unpreconditioned resid norm 7.885850510808e-08 true resid norm <br>
>> 7.885850531446e-08 ||r(i)||/||b|| 1.964110975313e-09<br>
>> 147 KSP unpreconditioned resid norm 6.483393946950e-08 true resid norm <br>
>> 6.483393931383e-08 ||r(i)||/||b|| 1.614804278515e-09<br>
>> 148 KSP unpreconditioned resid norm 5.690132597004e-08 true resid norm <br>
>> 5.690132577518e-08 ||r(i)||/||b|| 1.417228465328e-09<br>
>> 149 KSP unpreconditioned resid norm 5.023671521579e-08 true resid norm <br>
>> 5.023671502186e-08 ||r(i)||/||b|| 1.251234511035e-09<br>
>> 150 KSP unpreconditioned resid norm 4.625371062660e-08 true resid norm <br>
>> 4.625371062660e-08 ||r(i)||/||b|| 1.152030720445e-09<br>
>> 151 KSP unpreconditioned resid norm 4.349049084805e-08 true resid norm <br>
>> 4.349049089337e-08 ||r(i)||/||b|| 1.083207830846e-09<br>
>> 152 KSP unpreconditioned resid norm 3.932593324498e-08 true resid norm <br>
>> 3.932593376918e-08 ||r(i)||/||b|| 9.794821474546e-10<br>
>> 153 KSP unpreconditioned resid norm 3.504167649202e-08 true resid norm <br>
>> 3.504167638113e-08 ||r(i)||/||b|| 8.727751166356e-10<br>
>> 154 KSP unpreconditioned resid norm 2.892726347747e-08 true resid norm <br>
>> 2.892726348583e-08 ||r(i)||/||b|| 7.204848160858e-10<br>
>> 155 KSP unpreconditioned resid norm 2.477647033202e-08 true resid norm <br>
>> 2.477647041570e-08 ||r(i)||/||b|| 6.171019508795e-10<br>
>> 156 KSP unpreconditioned resid norm 2.128504065757e-08 true resid norm <br>
>> 2.128504067423e-08 ||r(i)||/||b|| 5.301416991298e-10<br>
>> 157 KSP unpreconditioned resid norm 1.879248809429e-08 true resid norm <br>
>> 1.879248818928e-08 ||r(i)||/||b|| 4.680602575310e-10<br>
>> 158 KSP unpreconditioned resid norm 1.673649140073e-08 true resid norm <br>
>> 1.673649134005e-08 ||r(i)||/||b|| 4.168520085200e-10<br>
>> 159 KSP unpreconditioned resid norm 1.497123388109e-08 true resid norm <br>
>> 1.497123365569e-08 ||r(i)||/||b|| 3.728851342016e-10<br>
>> 160 KSP unpreconditioned resid norm 1.315982130162e-08 true resid norm <br>
>> 1.315982149329e-08 ||r(i)||/||b|| 3.277687007261e-10<br>
>> 161 KSP unpreconditioned resid norm 1.182395864938e-08 true resid norm <br>
>> 1.182395868430e-08 ||r(i)||/||b|| 2.944966675550e-10<br>
>> 162 KSP unpreconditioned resid norm 1.070204481679e-08 true resid norm <br>
>> 1.070204466432e-08 ||r(i)||/||b|| 2.665534085342e-10<br>
>> 163 KSP unpreconditioned resid norm 9.969290307649e-09 true resid norm <br>
>> 9.969290432333e-09 ||r(i)||/||b|| 2.483028644297e-10<br>
>> 164 KSP unpreconditioned resid norm 9.134440883306e-09 true resid norm <br>
>> 9.134440980976e-09 ||r(i)||/||b|| 2.275094577628e-10<br>
>> 165 KSP unpreconditioned resid norm 8.593316427292e-09 true resid norm <br>
>> 8.593316413360e-09 ||r(i)||/||b|| 2.140317904139e-10<br>
>> 166 KSP unpreconditioned resid norm 8.042173048464e-09 true resid norm <br>
>> 8.042173332848e-09 ||r(i)||/||b|| 2.003045942277e-10<br>
>> 167 KSP unpreconditioned resid norm 7.655518522782e-09 true resid norm <br>
>> 7.655518879144e-09 ||r(i)||/||b|| 1.906742791064e-10<br>
>> 168 KSP unpreconditioned resid norm 7.210283391815e-09 true resid norm <br>
>> 7.210283220312e-09 ||r(i)||/||b|| 1.795848951442e-10<br>
>> 169 KSP unpreconditioned resid norm 6.793967416271e-09 true resid norm <br>
>> 6.793967448832e-09 ||r(i)||/||b|| 1.692158122825e-10<br>
>> 170 KSP unpreconditioned resid norm 6.249160304588e-09 true resid norm <br>
>> 6.249160382647e-09 ||r(i)||/||b|| 1.556464257736e-10<br>
>> 171 KSP unpreconditioned resid norm 5.794936438798e-09 true resid norm <br>
>> 5.794936332552e-09 ||r(i)||/||b|| 1.443331699811e-10<br>
>> 172 KSP unpreconditioned resid norm 5.222337397128e-09 true resid norm <br>
>> 5.222337443277e-09 ||r(i)||/||b|| 1.300715788135e-10<br>
>> 173 KSP unpreconditioned resid norm 4.755359110447e-09 true resid norm <br>
>> 4.755358888996e-09 ||r(i)||/||b|| 1.184406494668e-10<br>
>> 174 KSP unpreconditioned resid norm 4.317537007873e-09 true resid norm <br>
>> 4.317537267718e-09 ||r(i)||/||b|| 1.075359252630e-10<br>
>> 175 KSP unpreconditioned resid norm 3.924177535665e-09 true resid norm <br>
>> 3.924177629720e-09 ||r(i)||/||b|| 9.773860563138e-11<br>
>> 176 KSP unpreconditioned resid norm 3.502843065115e-09 true resid norm <br>
>> 3.502843126359e-09 ||r(i)||/||b|| 8.724452234855e-11<br>
>> 177 KSP unpreconditioned resid norm 3.083873232869e-09 true resid norm <br>
>> 3.083873352938e-09 ||r(i)||/||b|| 7.680933686007e-11<br>
>> 178 KSP unpreconditioned resid norm 2.758980676473e-09 true resid norm <br>
>> 2.758980618096e-09 ||r(i)||/||b|| 6.871730691658e-11<br>
>> 179 KSP unpreconditioned resid norm 2.510978240429e-09 true resid norm <br>
>> 2.510978327392e-09 ||r(i)||/||b|| 6.254036989334e-11<br>
>> 180 KSP unpreconditioned resid norm 2.323000193205e-09 true resid norm <br>
>> 2.323000193205e-09 ||r(i)||/||b|| 5.785844097519e-11<br>
>> 181 KSP unpreconditioned resid norm 2.167480159274e-09 true resid norm <br>
>> 2.167480113693e-09 ||r(i)||/||b|| 5.398493749153e-11<br>
>> 182 KSP unpreconditioned resid norm 1.983545827983e-09 true resid norm <br>
>> 1.983546404840e-09 ||r(i)||/||b|| 4.940374216139e-11<br>
>> 183 KSP unpreconditioned resid norm 1.794576286774e-09 true resid norm <br>
>> 1.794576224361e-09 ||r(i)||/||b|| 4.469710457036e-11<br>
>> 184 KSP unpreconditioned resid norm 1.583490590644e-09 true resid norm <br>
>> 1.583490380603e-09 ||r(i)||/||b|| 3.943963715064e-11<br>
>> 185 KSP unpreconditioned resid norm 1.412659866247e-09 true resid norm <br>
>> 1.412659832191e-09 ||r(i)||/||b|| 3.518479927722e-11<br>
>> 186 KSP unpreconditioned resid norm 1.285613344939e-09 true resid norm <br>
>> 1.285612984761e-09 ||r(i)||/||b|| 3.202047215205e-11<br>
>> 187 KSP unpreconditioned resid norm 1.168115133929e-09 true resid norm <br>
>> 1.168114766904e-09 ||r(i)||/||b|| 2.909397058634e-11<br>
>> 188 KSP unpreconditioned resid norm 1.063377926053e-09 true resid norm <br>
>> 1.063377647554e-09 ||r(i)||/||b|| 2.648530681802e-11<br>
>> 189 KSP unpreconditioned resid norm 9.548967728122e-10 true resid norm <br>
>> 9.548964523410e-10 ||r(i)||/||b|| 2.378339019807e-11<br>
>> KSP Object: 16 MPI processes<br>
>> type: fgmres<br>
>> restart=30, using Classical (unmodified) Gram-Schmidt <br>
>> Orthogonalization with no iterative refinement<br>
>> happy breakdown tolerance 1e-30<br>
>> maximum iterations=2000, initial guess is zero<br>
>> tolerances: relative=1e-20, absolute=1e-09, divergence=10000.<br>
>> right preconditioning<br>
>> using UNPRECONDITIONED norm type for convergence test<br>
>> PC Object: 16 MPI processes<br>
>> type: bjacobi<br>
>> number of blocks = 4<br>
>> Local solver information for first block is in the following KSP <br>
>> and PC objects on rank 0:<br>
>> Use -ksp_view ::ascii_info_detail to display information for all blocks<br>
>> KSP Object: (sub_) 4 MPI processes<br>
>> type: preonly<br>
>> maximum iterations=10000, initial guess is zero<br>
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>> left preconditioning<br>
>> using NONE norm type for convergence test<br>
>> PC Object: (sub_) 4 MPI processes<br>
>> type: telescope<br>
>> petsc subcomm: parent comm size reduction factor = 4<br>
>> petsc subcomm: parent_size = 4 , subcomm_size = 1<br>
>> petsc subcomm type = contiguous<br>
>> linear system matrix = precond matrix:<br>
>> Mat Object: (sub_) 4 MPI processes<br>
>> type: mpiaij<br>
>> rows=40200, cols=40200<br>
>> total: nonzeros=199996, allocated nonzeros=203412<br>
>> total number of mallocs used during MatSetValues calls=0<br>
>> not using I-node (on process 0) routines<br>
>> setup type: default<br>
>> Parent DM object: NULL<br>
>> Sub DM object: NULL<br>
>> KSP Object: (sub_telescope_) 1 MPI processes<br>
>> type: preonly<br>
>> maximum iterations=10000, initial guess is zero<br>
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>> left preconditioning<br>
>> using NONE norm type for convergence test<br>
>> PC Object: (sub_telescope_) 1 MPI processes<br>
>> type: lu<br>
>> out-of-place factorization<br>
>> tolerance for zero pivot 2.22045e-14<br>
>> matrix ordering: external<br>
>> factor fill ratio given 0., needed 0.<br>
>> Factored matrix follows:<br>
>> Mat Object: 1 MPI processes<br>
>> type: mumps<br>
>> rows=40200, cols=40200<br>
>> package used to perform factorization: mumps<br>
>> total: nonzeros=1849788, allocated nonzeros=1849788<br>
>> MUMPS run parameters:<br>
>> SYM (matrix type): 0<br>
>> PAR (host participation): 1<br>
>> ICNTL(1) (output for error): 6<br>
>> ICNTL(2) (output of diagnostic msg): 0<br>
>> ICNTL(3) (output for global info): 0<br>
>> ICNTL(4) (level of printing): 0<br>
>> ICNTL(5) (input mat struct): 0<br>
>> ICNTL(6) (matrix prescaling): 7<br>
>> ICNTL(7) (sequential matrix ordering):7<br>
>> ICNTL(8) (scaling strategy): 77<br>
>> ICNTL(10) (max num of refinements): 0<br>
>> ICNTL(11) (error analysis): 0<br>
>> ICNTL(12) (efficiency control): 1<br>
>> ICNTL(13) (sequential factorization of the root <br>
>> node): 0<br>
>> ICNTL(14) (percentage of estimated workspace <br>
>> increase): 20<br>
>> ICNTL(18) (input mat struct): 0<br>
>> ICNTL(19) (Schur complement info): 0<br>
>> ICNTL(20) (RHS sparse pattern): 0<br>
>> ICNTL(21) (solution struct): 0<br>
>> ICNTL(22) (in-core/out-of-core facility): 0<br>
>> ICNTL(23) (max size of memory can be allocated <br>
>> locally):0<br>
>> ICNTL(24) (detection of null pivot rows): 0<br>
>> ICNTL(25) (computation of a null space basis): <br>
>> 0<br>
>> ICNTL(26) (Schur options for RHS or solution): <br>
>> 0<br>
>> ICNTL(27) (blocking size for multiple RHS): <br>
>> -32<br>
>> ICNTL(28) (use parallel or sequential ordering): <br>
>> 1<br>
>> ICNTL(29) (parallel ordering): 0<br>
>> ICNTL(30) (user-specified set of entries in <br>
>> inv(A)): 0<br>
>> ICNTL(31) (factors is discarded in the solve <br>
>> phase): 0<br>
>> ICNTL(33) (compute determinant): 0<br>
>> ICNTL(35) (activate BLR based factorization): <br>
>> 0<br>
>> ICNTL(36) (choice of BLR factorization variant): <br>
>> 0<br>
>> ICNTL(38) (estimated compression rate of LU <br>
>> factors): 333<br>
>> CNTL(1) (relative pivoting threshold): 0.01<br>
>> CNTL(2) (stopping criterion of refinement): <br>
>> 1.49012e-08<br>
>> CNTL(3) (absolute pivoting threshold): 0.<br>
>> CNTL(4) (value of static pivoting): -1.<br>
>> CNTL(5) (fixation for null pivots): 0.<br>
>> CNTL(7) (dropping parameter for BLR): 0.<br>
>> RINFO(1) (local estimated flops for the <br>
>> elimination after analysis):<br>
>> [0] 1.45525e+08<br>
>> RINFO(2) (local estimated flops for the assembly <br>
>> after factorization):<br>
>> [0] 2.89397e+06<br>
>> RINFO(3) (local estimated flops for the <br>
>> elimination after factorization):<br>
>> [0] 1.45525e+08<br>
>> INFO(15) (estimated size of (in MB) MUMPS <br>
>> internal data for running numerical factorization):<br>
>> [0] 29<br>
>> INFO(16) (size of (in MB) MUMPS internal data <br>
>> used during numerical factorization):<br>
>> [0] 29<br>
>> INFO(23) (num of pivots eliminated on this <br>
>> processor after factorization):<br>
>> [0] 40200<br>
>> RINFOG(1) (global estimated flops for the <br>
>> elimination after analysis): 1.45525e+08<br>
>> RINFOG(2) (global estimated flops for the <br>
>> assembly after factorization): 2.89397e+06<br>
>> RINFOG(3) (global estimated flops for the <br>
>> elimination after factorization): 1.45525e+08<br>
>> (RINFOG(12) RINFOG(13))*2^INFOG(34) <br>
>> (determinant): (0.,0.)*(2^0)<br>
>> INFOG(3) (estimated real workspace for factors on <br>
>> all processors after analysis): 1849788<br>
>> INFOG(4) (estimated integer workspace for factors <br>
>> on all processors after analysis): 879986<br>
>> INFOG(5) (estimated maximum front size in the <br>
>> complete tree): 282<br>
>> INFOG(6) (number of nodes in the complete tree): <br>
>> 23709<br>
>> INFOG(7) (ordering option effectively used after <br>
>> analysis): 5<br>
>> INFOG(8) (structural symmetry in percent of the <br>
>> permuted matrix after analysis): 100<br>
>> INFOG(9) (total real/complex workspace to store <br>
>> the matrix factors after factorization): 1849788<br>
>> INFOG(10) (total integer space store the matrix <br>
>> factors after factorization): 879986<br>
>> INFOG(11) (order of largest frontal matrix after <br>
>> factorization): 282<br>
>> INFOG(12) (number of off-diagonal pivots): 0<br>
>> INFOG(13) (number of delayed pivots after <br>
>> factorization): 0<br>
>> INFOG(14) (number of memory compress after <br>
>> factorization): 0<br>
>> INFOG(15) (number of steps of iterative <br>
>> refinement after solution): 0<br>
>> INFOG(16) (estimated size (in MB) of all MUMPS <br>
>> internal data for factorization after analysis: value on the most <br>
>> memory consuming processor): 29<br>
>> INFOG(17) (estimated size of all MUMPS internal <br>
>> data for factorization after analysis: sum over all processors): 29<br>
>> INFOG(18) (size of all MUMPS internal data <br>
>> allocated during factorization: value on the most memory consuming <br>
>> processor): 29<br>
>> INFOG(19) (size of all MUMPS internal data <br>
>> allocated during factorization: sum over all processors): 29<br>
>> INFOG(20) (estimated number of entries in the <br>
>> factors): 1849788<br>
>> INFOG(21) (size in MB of memory effectively used <br>
>> during factorization - value on the most memory consuming processor): 26<br>
>> INFOG(22) (size in MB of memory effectively used <br>
>> during factorization - sum over all processors): 26<br>
>> INFOG(23) (after analysis: value of ICNTL(6) <br>
>> effectively used): 0<br>
>> INFOG(24) (after analysis: value of ICNTL(12) <br>
>> effectively used): 1<br>
>> INFOG(25) (after factorization: number of pivots <br>
>> modified by static pivoting): 0<br>
>> INFOG(28) (after factorization: number of null <br>
>> pivots encountered): 0<br>
>> INFOG(29) (after factorization: effective number <br>
>> of entries in the factors (sum over all processors)): 1849788<br>
>> INFOG(30, 31) (after solution: size in Mbytes of <br>
>> memory used during solution phase): 29, 29<br>
>> INFOG(32) (after analysis: type of analysis done): 1<br>
>> INFOG(33) (value used for ICNTL(8)): 7<br>
>> INFOG(34) (exponent of the determinant if <br>
>> determinant is requested): 0<br>
>> INFOG(35) (after factorization: number of entries <br>
>> taking into account BLR factor compression - sum over all processors): <br>
>> 1849788<br>
>> INFOG(36) (after analysis: estimated size of all <br>
>> MUMPS internal data for running BLR in-core - value on the most memory <br>
>> consuming processor): 0<br>
>> INFOG(37) (after analysis: estimated size of all <br>
>> MUMPS internal data for running BLR in-core - sum over all processors): 0<br>
>> INFOG(38) (after analysis: estimated size of all <br>
>> MUMPS internal data for running BLR out-of-core - value on the most <br>
>> memory consuming processor): 0<br>
>> INFOG(39) (after analysis: estimated size of all <br>
>> MUMPS internal data for running BLR out-of-core - sum over all <br>
>> processors): 0<br>
>> linear system matrix = precond matrix:<br>
>> Mat Object: 1 MPI processes<br>
>> type: seqaijcusparse<br>
>> rows=40200, cols=40200<br>
>> total: nonzeros=199996, allocated nonzeros=199996<br>
>> total number of mallocs used during MatSetValues calls=0<br>
>> not using I-node routines<br>
>> linear system matrix = precond matrix:<br>
>> Mat Object: 16 MPI processes<br>
>> type: mpiaijcusparse<br>
>> rows=160800, cols=160800<br>
>> total: nonzeros=802396, allocated nonzeros=1608000<br>
>> total number of mallocs used during MatSetValues calls=0<br>
>> not using I-node (on process 0) routines<br>
>> Norm of error 9.11684e-07 iterations 189<br>
>><br>
>> Chang<br>
>><br>
>><br>
>><br>
>> On 10/14/21 10:10 PM, Chang Liu wrote:<br>
>>> Hi Barry,<br>
>>> No problem. Here is the output. It seems that the resid norm <br>
>>> calculation is incorrect.<br>
>>> $ mpiexec -n 16 --hostfile hostfile --oversubscribe ./ex7 -m 400 <br>
>>> -ksp_view -ksp_monitor_true_residual -pc_type bjacobi <br>
>>> -pc_bjacobi_blocks 4 -ksp_type fgmres -mat_type aijcusparse <br>
>>> -sub_pc_type telescope -sub_ksp_type preonly -sub_telescope_ksp_type <br>
>>> preonly -sub_telescope_pc_type lu <br>
>>> -sub_telescope_pc_factor_mat_solver_type cusparse <br>
>>> -sub_pc_telescope_reduction_factor 4 -sub_pc_telescope_subcomm_type <br>
>>> contiguous -ksp_max_it 2000 -ksp_rtol 1.e-20 -ksp_atol 1.e-9<br>
>>> 0 KSP unpreconditioned resid norm 4.014971979977e+01 true resid <br>
>>> norm 4.014971979977e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
>>> 1 KSP unpreconditioned resid norm 0.000000000000e+00 true resid <br>
>>> norm 4.014971979977e+01 ||r(i)||/||b|| 1.000000000000e+00<br>
>>> KSP Object: 16 MPI processes<br>
>>> type: fgmres<br>
>>> restart=30, using Classical (unmodified) Gram-Schmidt <br>
>>> Orthogonalization with no iterative refinement<br>
>>> happy breakdown tolerance 1e-30<br>
>>> maximum iterations=2000, initial guess is zero<br>
>>> tolerances: relative=1e-20, absolute=1e-09, divergence=10000.<br>
>>> right preconditioning<br>
>>> using UNPRECONDITIONED norm type for convergence test<br>
>>> PC Object: 16 MPI processes<br>
>>> type: bjacobi<br>
>>> number of blocks = 4<br>
>>> Local solver information for first block is in the following KSP <br>
>>> and PC objects on rank 0:<br>
>>> Use -ksp_view ::ascii_info_detail to display information for all <br>
>>> blocks<br>
>>> KSP Object: (sub_) 4 MPI processes<br>
>>> type: preonly<br>
>>> maximum iterations=10000, initial guess is zero<br>
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>>> left preconditioning<br>
>>> using NONE norm type for convergence test<br>
>>> PC Object: (sub_) 4 MPI processes<br>
>>> type: telescope<br>
>>> petsc subcomm: parent comm size reduction factor = 4<br>
>>> petsc subcomm: parent_size = 4 , subcomm_size = 1<br>
>>> petsc subcomm type = contiguous<br>
>>> linear system matrix = precond matrix:<br>
>>> Mat Object: (sub_) 4 MPI processes<br>
>>> type: mpiaij<br>
>>> rows=40200, cols=40200<br>
>>> total: nonzeros=199996, allocated nonzeros=203412<br>
>>> total number of mallocs used during MatSetValues calls=0<br>
>>> not using I-node (on process 0) routines<br>
>>> setup type: default<br>
>>> Parent DM object: NULL<br>
>>> Sub DM object: NULL<br>
>>> KSP Object: (sub_telescope_) 1 MPI processes<br>
>>> type: preonly<br>
>>> maximum iterations=10000, initial guess is zero<br>
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>>> left preconditioning<br>
>>> using NONE norm type for convergence test<br>
>>> PC Object: (sub_telescope_) 1 MPI processes<br>
>>> type: lu<br>
>>> out-of-place factorization<br>
>>> tolerance for zero pivot 2.22045e-14<br>
>>> matrix ordering: nd<br>
>>> factor fill ratio given 5., needed 8.62558<br>
>>> Factored matrix follows:<br>
>>> Mat Object: 1 MPI processes<br>
>>> type: seqaijcusparse<br>
>>> rows=40200, cols=40200<br>
>>> package used to perform factorization: cusparse<br>
>>> total: nonzeros=1725082, allocated nonzeros=1725082<br>
>>> not using I-node routines<br>
>>> linear system matrix = precond matrix:<br>
>>> Mat Object: 1 MPI processes<br>
>>> type: seqaijcusparse<br>
>>> rows=40200, cols=40200<br>
>>> total: nonzeros=199996, allocated nonzeros=199996<br>
>>> total number of mallocs used during MatSetValues calls=0<br>
>>> not using I-node routines<br>
>>> linear system matrix = precond matrix:<br>
>>> Mat Object: 16 MPI processes<br>
>>> type: mpiaijcusparse<br>
>>> rows=160800, cols=160800<br>
>>> total: nonzeros=802396, allocated nonzeros=1608000<br>
>>> total number of mallocs used during MatSetValues calls=0<br>
>>> not using I-node (on process 0) routines<br>
>>> Norm of error 400.999 iterations 1<br>
>>> Chang<br>
>>> On 10/14/21 9:47 PM, Barry Smith wrote:<br>
>>>><br>
>>>> Chang,<br>
>>>><br>
>>>> Sorry I did not notice that one. Please run that with -ksp_view <br>
>>>> -ksp_monitor_true_residual so we can see exactly how options are <br>
>>>> interpreted and solver used. At a glance it looks ok but something <br>
>>>> must be wrong to get the wrong answer.<br>
>>>><br>
>>>> Barry<br>
>>>><br>
>>>>> On Oct 14, 2021, at 6:02 PM, Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> wrote:<br>
>>>>><br>
>>>>> Hi Barry,<br>
>>>>><br>
>>>>> That is exactly what I was doing in the second example, in which <br>
>>>>> the preconditioner works but the GMRES does not.<br>
>>>>><br>
>>>>> Chang<br>
>>>>><br>
>>>>> On 10/14/21 5:15 PM, Barry Smith wrote:<br>
>>>>>> You need to use the PCTELESCOPE inside the block Jacobi, not <br>
>>>>>> outside it. So something like -pc_type bjacobi -sub_pc_type <br>
>>>>>> telescope -sub_telescope_pc_type lu<br>
>>>>>>> On Oct 14, 2021, at 4:14 PM, Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> wrote:<br>
>>>>>>><br>
>>>>>>> Hi Pierre,<br>
>>>>>>><br>
>>>>>>> I wonder if the trick of PCTELESCOPE only works for <br>
>>>>>>> preconditioner and not for the solver. I have done some tests, <br>
>>>>>>> and find that for solving a small matrix using <br>
>>>>>>> -telescope_ksp_type preonly, it does work for GPU with multiple <br>
>>>>>>> MPI processes. However, for bjacobi and gmres, it does not work.<br>
>>>>>>><br>
>>>>>>> The command line options I used for small matrix is like<br>
>>>>>>><br>
>>>>>>> mpiexec -n 4 --oversubscribe ./ex7 -m 100 -ksp_monitor_short <br>
>>>>>>> -pc_type telescope -mat_type aijcusparse -telescope_pc_type lu <br>
>>>>>>> -telescope_pc_factor_mat_solver_type cusparse -telescope_ksp_type <br>
>>>>>>> preonly -pc_telescope_reduction_factor 4<br>
>>>>>>><br>
>>>>>>> which gives the correct output. For iterative solver, I tried<br>
>>>>>>><br>
>>>>>>> mpiexec -n 16 --oversubscribe ./ex7 -m 400 -ksp_monitor_short <br>
>>>>>>> -pc_type bjacobi -pc_bjacobi_blocks 4 -ksp_type fgmres -mat_type <br>
>>>>>>> aijcusparse -sub_pc_type telescope -sub_ksp_type preonly <br>
>>>>>>> -sub_telescope_ksp_type preonly -sub_telescope_pc_type lu <br>
>>>>>>> -sub_telescope_pc_factor_mat_solver_type cusparse <br>
>>>>>>> -sub_pc_telescope_reduction_factor 4 -ksp_max_it 2000 -ksp_rtol <br>
>>>>>>> 1.e-9 -ksp_atol 1.e-20<br>
>>>>>>><br>
>>>>>>> for large matrix. The output is like<br>
>>>>>>><br>
>>>>>>> 0 KSP Residual norm 40.1497<br>
>>>>>>> 1 KSP Residual norm < 1.e-11<br>
>>>>>>> Norm of error 400.999 iterations 1<br>
>>>>>>><br>
>>>>>>> So it seems to call a direct solver instead of an iterative one.<br>
>>>>>>><br>
>>>>>>> Can you please help check these options?<br>
>>>>>>><br>
>>>>>>> Chang<br>
>>>>>>><br>
>>>>>>> On 10/14/21 10:04 AM, Pierre Jolivet wrote:<br>
>>>>>>>>> On 14 Oct 2021, at 3:50 PM, Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> Thank you Pierre. I was not aware of PCTELESCOPE before. This <br>
>>>>>>>>> sounds exactly what I need. I wonder if PCTELESCOPE can <br>
>>>>>>>>> transform a mpiaijcusparse to seqaircusparse? Or I have to do <br>
>>>>>>>>> it manually?<br>
>>>>>>>> PCTELESCOPE uses MatCreateMPIMatConcatenateSeqMat().<br>
>>>>>>>> 1) I’m not sure this is implemented for cuSparse matrices, but <br>
>>>>>>>> it should be;<br>
>>>>>>>> 2) at least for the implementations <br>
>>>>>>>> MatCreateMPIMatConcatenateSeqMat_MPIBAIJ() and <br>
>>>>>>>> MatCreateMPIMatConcatenateSeqMat_MPIAIJ(), the resulting MatType <br>
>>>>>>>> is MATBAIJ (resp. MATAIJ). Constructors are usually “smart” <br>
>>>>>>>> enough to detect if the MPI communicator on which the Mat lives <br>
>>>>>>>> is of size 1 (your case), and then the resulting Mat is of type <br>
>>>>>>>> MatSeqX instead of MatMPIX, so you would not need to worry about <br>
>>>>>>>> the transformation you are mentioning.<br>
>>>>>>>> If you try this out and this does not work, please provide the <br>
>>>>>>>> backtrace (probably something like “Operation XYZ not <br>
>>>>>>>> implemented for MatType ABC”), and hopefully someone can add the <br>
>>>>>>>> missing plumbing.<br>
>>>>>>>> I do not claim that this will be efficient, but I think this <br>
>>>>>>>> goes in the direction of what you want to achieve.<br>
>>>>>>>> Thanks,<br>
>>>>>>>> Pierre<br>
>>>>>>>>> Chang<br>
>>>>>>>>><br>
>>>>>>>>> On 10/14/21 1:35 AM, Pierre Jolivet wrote:<br>
>>>>>>>>>> Maybe I’m missing something, but can’t you use PCTELESCOPE as <br>
>>>>>>>>>> a subdomain solver, with a reduction factor equal to the <br>
>>>>>>>>>> number of MPI processes you have per block?<br>
>>>>>>>>>> -sub_pc_type telescope -sub_pc_telescope_reduction_factor X <br>
>>>>>>>>>> -sub_telescope_pc_type lu<br>
>>>>>>>>>> This does not work with MUMPS -mat_mumps_use_omp_threads <br>
>>>>>>>>>> because not only do the Mat needs to be redistributed, the <br>
>>>>>>>>>> secondary processes also need to be “converted” to OpenMP threads.<br>
>>>>>>>>>> Thus the need for specific code in mumps.c.<br>
>>>>>>>>>> Thanks,<br>
>>>>>>>>>> Pierre<br>
>>>>>>>>>>> On 14 Oct 2021, at 6:00 AM, Chang Liu via petsc-users <br>
>>>>>>>>>>> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>> wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi Junchao,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Yes that is what I want.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Chang<br>
>>>>>>>>>>><br>
>>>>>>>>>>> On 10/13/21 11:42 PM, Junchao Zhang wrote:<br>
>>>>>>>>>>>> On Wed, Oct 13, 2021 at 8:58 PM Barry Smith <br>
>>>>>>>>>>>> <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a> <mailto:<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a> <mailto:<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>>> wrote:<br>
>>>>>>>>>>>> Junchao,<br>
>>>>>>>>>>>> If I understand correctly Chang is using the block <br>
>>>>>>>>>>>> Jacobi<br>
>>>>>>>>>>>> method with a single block for a number of MPI ranks and <br>
>>>>>>>>>>>> a direct<br>
>>>>>>>>>>>> solver for each block so it uses <br>
>>>>>>>>>>>> PCSetUp_BJacobi_Multiproc() which<br>
>>>>>>>>>>>> is code Hong Zhang wrote a number of years ago for CPUs. <br>
>>>>>>>>>>>> For their<br>
>>>>>>>>>>>> particular problems this preconditioner works well, but <br>
>>>>>>>>>>>> using an<br>
>>>>>>>>>>>> iterative solver on the blocks does not work well.<br>
>>>>>>>>>>>> If we had complete MPI-GPU direct solvers he could <br>
>>>>>>>>>>>> just use<br>
>>>>>>>>>>>> the current code with MPIAIJCUSPARSE on each block but <br>
>>>>>>>>>>>> since we do<br>
>>>>>>>>>>>> not he would like to use a single GPU for each block, <br>
>>>>>>>>>>>> this means<br>
>>>>>>>>>>>> that diagonal blocks of the global parallel MPI matrix <br>
>>>>>>>>>>>> needs to be<br>
>>>>>>>>>>>> sent to a subset of the GPUs (one GPU per block, which <br>
>>>>>>>>>>>> has multiple<br>
>>>>>>>>>>>> MPI ranks associated with the blocks). Similarly for the <br>
>>>>>>>>>>>> triangular<br>
>>>>>>>>>>>> solves the blocks of the right hand side needs to be <br>
>>>>>>>>>>>> shipped to the<br>
>>>>>>>>>>>> appropriate GPU and the resulting solution shipped back <br>
>>>>>>>>>>>> to the<br>
>>>>>>>>>>>> multiple GPUs. So Chang is absolutely correct, this is <br>
>>>>>>>>>>>> somewhat like<br>
>>>>>>>>>>>> your code for MUMPS with OpenMP. OK, I now understand <br>
>>>>>>>>>>>> the background..<br>
>>>>>>>>>>>> One could use PCSetUp_BJacobi_Multiproc() and get the <br>
>>>>>>>>>>>> blocks on the<br>
>>>>>>>>>>>> MPI ranks and then shrink each block down to a single <br>
>>>>>>>>>>>> GPU but this<br>
>>>>>>>>>>>> would be pretty inefficient, ideally one would go <br>
>>>>>>>>>>>> directly from the<br>
>>>>>>>>>>>> big MPI matrix on all the GPUs to the sub matrices on <br>
>>>>>>>>>>>> the subset of<br>
>>>>>>>>>>>> GPUs. But this may be a large coding project.<br>
>>>>>>>>>>>> I don't understand these sentences. Why do you say "shrink"? <br>
>>>>>>>>>>>> In my mind, we just need to move each block (submatrix) <br>
>>>>>>>>>>>> living over multiple MPI ranks to one of them and solve <br>
>>>>>>>>>>>> directly there. In other words, we keep blocks' size, no <br>
>>>>>>>>>>>> shrinking or expanding.<br>
>>>>>>>>>>>> As mentioned before, cusparse does not provide LU <br>
>>>>>>>>>>>> factorization. So the LU factorization would be done on CPU, <br>
>>>>>>>>>>>> and the solve be done on GPU. I assume Chang wants to gain <br>
>>>>>>>>>>>> from the (potential) faster solve (instead of factorization) <br>
>>>>>>>>>>>> on GPU.<br>
>>>>>>>>>>>> Barry<br>
>>>>>>>>>>>> Since the matrices being factored and solved directly <br>
>>>>>>>>>>>> are relatively<br>
>>>>>>>>>>>> large it is possible that the cusparse code could be <br>
>>>>>>>>>>>> reasonably<br>
>>>>>>>>>>>> efficient (they are not the tiny problems one gets at <br>
>>>>>>>>>>>> the coarse<br>
>>>>>>>>>>>> level of multigrid). Of course, this is speculation, I don't<br>
>>>>>>>>>>>> actually know how much better the cusparse code would be <br>
>>>>>>>>>>>> on the<br>
>>>>>>>>>>>> direct solver than a good CPU direct sparse solver.<br>
>>>>>>>>>>>> > On Oct 13, 2021, at 9:32 PM, Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>> wrote:<br>
>>>>>>>>>>>> ><br>
>>>>>>>>>>>> > Sorry I am not familiar with the details either. Can <br>
>>>>>>>>>>>> you please<br>
>>>>>>>>>>>> check the code in MatMumpsGatherNonzerosOnMaster in mumps.c?<br>
>>>>>>>>>>>> ><br>
>>>>>>>>>>>> > Chang<br>
>>>>>>>>>>>> ><br>
>>>>>>>>>>>> > On 10/13/21 9:24 PM, Junchao Zhang wrote:<br>
>>>>>>>>>>>> >> Hi Chang,<br>
>>>>>>>>>>>> >> I did the work in mumps. It is easy for me to <br>
>>>>>>>>>>>> understand<br>
>>>>>>>>>>>> gathering matrix rows to one process.<br>
>>>>>>>>>>>> >> But how to gather blocks (submatrices) to form a <br>
>>>>>>>>>>>> large block? Can you draw a picture of that?<br>
>>>>>>>>>>>> >> Thanks<br>
>>>>>>>>>>>> >> --Junchao Zhang<br>
>>>>>>>>>>>> >> On Wed, Oct 13, 2021 at 7:47 PM Chang Liu via <br>
>>>>>>>>>>>> petsc-users<br>
>>>>>>>>>>>> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> wrote:<br>
>>>>>>>>>>>> >> Hi Barry,<br>
>>>>>>>>>>>> >> I think mumps solver in petsc does support that. <br>
>>>>>>>>>>>> You can<br>
>>>>>>>>>>>> check the<br>
>>>>>>>>>>>> >> documentation on "-mat_mumps_use_omp_threads" at<br>
>>>>>>>>>>>> >><br>
>>>>>>>>>>>> <a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a> <br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a>> <br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a> <br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a>>> <br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> >> <br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a> <br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a>> <br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a> <br>
>>>>>>>>>>>> <<a href="https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html" rel="noreferrer" target="_blank">https://petsc.org/release/docs/manualpages/Mat/MATSOLVERMUMPS.html</a>>>> <br>
>>>>>>>>>>>><br>
>>>>>>>>>>>> >> and the code enclosed by #if<br>
>>>>>>>>>>>> defined(PETSC_HAVE_OPENMP_SUPPORT) in<br>
>>>>>>>>>>>> >> functions MatMumpsSetUpDistRHSInfo and<br>
>>>>>>>>>>>> >> MatMumpsGatherNonzerosOnMaster in<br>
>>>>>>>>>>>> >> mumps.c<br>
>>>>>>>>>>>> >> 1. I understand it is ideal to do one MPI rank <br>
>>>>>>>>>>>> per GPU.<br>
>>>>>>>>>>>> However, I am<br>
>>>>>>>>>>>> >> working on an existing code that was developed <br>
>>>>>>>>>>>> based on MPI<br>
>>>>>>>>>>>> and the the<br>
>>>>>>>>>>>> >> # of mpi ranks is typically equal to # of cpu <br>
>>>>>>>>>>>> cores. We don't<br>
>>>>>>>>>>>> want to<br>
>>>>>>>>>>>> >> change the whole structure of the code.<br>
>>>>>>>>>>>> >> 2. What you have suggested has been coded in <br>
>>>>>>>>>>>> mumps.c. See<br>
>>>>>>>>>>>> function<br>
>>>>>>>>>>>> >> MatMumpsSetUpDistRHSInfo.<br>
>>>>>>>>>>>> >> Regards,<br>
>>>>>>>>>>>> >> Chang<br>
>>>>>>>>>>>> >> On 10/13/21 7:53 PM, Barry Smith wrote:<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> >> On Oct 13, 2021, at 3:50 PM, Chang Liu <br>
>>>>>>>>>>>> <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>> wrote:<br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> Hi Barry,<br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> That is exactly what I want.<br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> Back to my original question, I am looking <br>
>>>>>>>>>>>> for an approach to<br>
>>>>>>>>>>>> >> transfer<br>
>>>>>>>>>>>> >> >> matrix<br>
>>>>>>>>>>>> >> >> data from many MPI processes to "master" MPI<br>
>>>>>>>>>>>> >> >> processes, each of which taking care of one <br>
>>>>>>>>>>>> GPU, and then<br>
>>>>>>>>>>>> upload<br>
>>>>>>>>>>>> >> the data to GPU to<br>
>>>>>>>>>>>> >> >> solve.<br>
>>>>>>>>>>>> >> >> One can just grab some codes from mumps.c to<br>
>>>>>>>>>>>> <a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">aijcusparse.cu</a> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a> <br>
>>>>>>>>>>>> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a>>><br>
>>>>>>>>>>>> >> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a>> <br>
>>>>>>>>>>>> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a> <<a href="http://aijcusparse.cu" rel="noreferrer" target="_blank">http://aijcusparse.cu</a>>>>.<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > mumps.c doesn't actually do that. It never <br>
>>>>>>>>>>>> needs to<br>
>>>>>>>>>>>> copy the<br>
>>>>>>>>>>>> >> entire matrix to a single MPI rank.<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > It would be possible to write such a code <br>
>>>>>>>>>>>> that you<br>
>>>>>>>>>>>> suggest but<br>
>>>>>>>>>>>> >> it is not clear that it makes sense<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > 1) For normal PETSc GPU usage there is one <br>
>>>>>>>>>>>> GPU per MPI<br>
>>>>>>>>>>>> rank, so<br>
>>>>>>>>>>>> >> while your one GPU per big domain is solving its <br>
>>>>>>>>>>>> systems the<br>
>>>>>>>>>>>> other<br>
>>>>>>>>>>>> >> GPUs (with the other MPI ranks that share that <br>
>>>>>>>>>>>> domain) are doing<br>
>>>>>>>>>>>> >> nothing.<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > 2) For each triangular solve you would have to <br>
>>>>>>>>>>>> gather the<br>
>>>>>>>>>>>> right<br>
>>>>>>>>>>>> >> hand side from the multiple ranks to the single <br>
>>>>>>>>>>>> GPU to pass it to<br>
>>>>>>>>>>>> >> the GPU solver and then scatter the resulting <br>
>>>>>>>>>>>> solution back<br>
>>>>>>>>>>>> to all<br>
>>>>>>>>>>>> >> of its subdomain ranks.<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > What I was suggesting was assign an entire <br>
>>>>>>>>>>>> subdomain to a<br>
>>>>>>>>>>>> >> single MPI rank, thus it does everything on one <br>
>>>>>>>>>>>> GPU and can<br>
>>>>>>>>>>>> use the<br>
>>>>>>>>>>>> >> GPU solver directly. If all the major <br>
>>>>>>>>>>>> computations of a subdomain<br>
>>>>>>>>>>>> >> can fit and be done on a single GPU then you would be<br>
>>>>>>>>>>>> utilizing all<br>
>>>>>>>>>>>> >> the GPUs you are using effectively.<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> > Barry<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> Chang<br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> On 10/13/21 1:53 PM, Barry Smith wrote:<br>
>>>>>>>>>>>> >> >>> Chang,<br>
>>>>>>>>>>>> >> >>> You are correct there is no MPI + GPU <br>
>>>>>>>>>>>> direct<br>
>>>>>>>>>>>> solvers that<br>
>>>>>>>>>>>> >> currently do the triangular solves with MPI + GPU <br>
>>>>>>>>>>>> parallelism<br>
>>>>>>>>>>>> that I<br>
>>>>>>>>>>>> >> am aware of. You are limited that individual <br>
>>>>>>>>>>>> triangular solves be<br>
>>>>>>>>>>>> >> done on a single GPU. I can only suggest making <br>
>>>>>>>>>>>> each subdomain as<br>
>>>>>>>>>>>> >> big as possible to utilize each GPU as much as <br>
>>>>>>>>>>>> possible for the<br>
>>>>>>>>>>>> >> direct triangular solves.<br>
>>>>>>>>>>>> >> >>> Barry<br>
>>>>>>>>>>>> >> >>>> On Oct 13, 2021, at 12:16 PM, Chang Liu via <br>
>>>>>>>>>>>> petsc-users<br>
>>>>>>>>>>>> >> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> wrote:<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> Hi Mark,<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> '-mat_type aijcusparse' works with <br>
>>>>>>>>>>>> mpiaijcusparse with<br>
>>>>>>>>>>>> other<br>
>>>>>>>>>>>> >> solvers, but with -pc_factor_mat_solver_type <br>
>>>>>>>>>>>> cusparse, it<br>
>>>>>>>>>>>> will give<br>
>>>>>>>>>>>> >> an error.<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> Yes what I want is to have mumps or superlu <br>
>>>>>>>>>>>> to do the<br>
>>>>>>>>>>>> >> factorization, and then do the rest, including <br>
>>>>>>>>>>>> GMRES solver,<br>
>>>>>>>>>>>> on gpu.<br>
>>>>>>>>>>>> >> Is that possible?<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> I have tried to use aijcusparse with <br>
>>>>>>>>>>>> superlu_dist, it<br>
>>>>>>>>>>>> runs but<br>
>>>>>>>>>>>> >> the iterative solver is still running on CPUs. I have<br>
>>>>>>>>>>>> contacted the<br>
>>>>>>>>>>>> >> superlu group and they confirmed that is the case <br>
>>>>>>>>>>>> right now.<br>
>>>>>>>>>>>> But if<br>
>>>>>>>>>>>> >> I set -pc_factor_mat_solver_type cusparse, it <br>
>>>>>>>>>>>> seems that the<br>
>>>>>>>>>>>> >> iterative solver is running on GPU.<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> Chang<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> On 10/13/21 12:03 PM, Mark Adams wrote:<br>
>>>>>>>>>>>> >> >>>>> On Wed, Oct 13, 2021 at 11:10 AM Chang Liu<br>
>>>>>>>>>>>> <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>> wrote:<br>
>>>>>>>>>>>> >> >>>>> Thank you Junchao for explaining this. <br>
>>>>>>>>>>>> I guess in<br>
>>>>>>>>>>>> my case<br>
>>>>>>>>>>>> >> the code is<br>
>>>>>>>>>>>> >> >>>>> just calling a seq solver like superlu <br>
>>>>>>>>>>>> to do<br>
>>>>>>>>>>>> >> factorization on GPUs.<br>
>>>>>>>>>>>> >> >>>>> My idea is that I want to have a <br>
>>>>>>>>>>>> traditional MPI<br>
>>>>>>>>>>>> code to<br>
>>>>>>>>>>>> >> utilize GPUs<br>
>>>>>>>>>>>> >> >>>>> with cusparse. Right now cusparse does <br>
>>>>>>>>>>>> not support<br>
>>>>>>>>>>>> mpiaij<br>
>>>>>>>>>>>> >> matrix, Sure it does: '-mat_type aijcusparse' <br>
>>>>>>>>>>>> will give you an<br>
>>>>>>>>>>>> >> mpiaijcusparse matrix with > 1 processes.<br>
>>>>>>>>>>>> >> >>>>> (-mat_type mpiaijcusparse might also work <br>
>>>>>>>>>>>> with >1 proc).<br>
>>>>>>>>>>>> >> >>>>> However, I see in grepping the repo that <br>
>>>>>>>>>>>> all the mumps and<br>
>>>>>>>>>>>> >> superlu tests use aij or sell matrix type.<br>
>>>>>>>>>>>> >> >>>>> MUMPS and SuperLU provide their own <br>
>>>>>>>>>>>> solves, I assume<br>
>>>>>>>>>>>> .... but<br>
>>>>>>>>>>>> >> you might want to do other matrix operations on <br>
>>>>>>>>>>>> the GPU. Is<br>
>>>>>>>>>>>> that the<br>
>>>>>>>>>>>> >> issue?<br>
>>>>>>>>>>>> >> >>>>> Did you try -mat_type aijcusparse with <br>
>>>>>>>>>>>> MUMPS and/or<br>
>>>>>>>>>>>> SuperLU<br>
>>>>>>>>>>>> >> have a problem? (no test with it so it probably <br>
>>>>>>>>>>>> does not work)<br>
>>>>>>>>>>>> >> >>>>> Thanks,<br>
>>>>>>>>>>>> >> >>>>> Mark<br>
>>>>>>>>>>>> >> >>>>> so I<br>
>>>>>>>>>>>> >> >>>>> want the code to have a mpiaij matrix <br>
>>>>>>>>>>>> when adding<br>
>>>>>>>>>>>> all the<br>
>>>>>>>>>>>> >> matrix terms,<br>
>>>>>>>>>>>> >> >>>>> and then transform the matrix to <br>
>>>>>>>>>>>> seqaij when doing the<br>
>>>>>>>>>>>> >> factorization<br>
>>>>>>>>>>>> >> >>>>> and<br>
>>>>>>>>>>>> >> >>>>> solve. This involves sending the data <br>
>>>>>>>>>>>> to the master<br>
>>>>>>>>>>>> >> process, and I<br>
>>>>>>>>>>>> >> >>>>> think<br>
>>>>>>>>>>>> >> >>>>> the petsc mumps solver have something <br>
>>>>>>>>>>>> similar already.<br>
>>>>>>>>>>>> >> >>>>> Chang<br>
>>>>>>>>>>>> >> >>>>> On 10/13/21 10:18 AM, Junchao Zhang wrote:<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > On Tue, Oct 12, 2021 at 1:07 PM <br>
>>>>>>>>>>>> Mark Adams<br>
>>>>>>>>>>>> >> <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>>>>>> wrote:<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > On Tue, Oct 12, 2021 at 1:45 PM <br>
>>>>>>>>>>>> Chang Liu<br>
>>>>>>>>>>>> >> <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>>> wrote:<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > Hi Mark,<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > The option I use is like<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > -pc_type bjacobi <br>
>>>>>>>>>>>> -pc_bjacobi_blocks 16<br>
>>>>>>>>>>>> >> -ksp_type fgmres<br>
>>>>>>>>>>>> >> >>>>> -mat_type<br>
>>>>>>>>>>>> >> >>>>> > aijcusparse <br>
>>>>>>>>>>>> *-sub_pc_factor_mat_solver_type<br>
>>>>>>>>>>>> >> cusparse<br>
>>>>>>>>>>>> >> >>>>> *-sub_ksp_type<br>
>>>>>>>>>>>> >> >>>>> > preonly *-sub_pc_type lu* <br>
>>>>>>>>>>>> -ksp_max_it 2000<br>
>>>>>>>>>>>> >> -ksp_rtol 1.e-300<br>
>>>>>>>>>>>> >> >>>>> > -ksp_atol 1.e-300<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > Note, If you use -log_view the <br>
>>>>>>>>>>>> last column<br>
>>>>>>>>>>>> (rows<br>
>>>>>>>>>>>> >> are the<br>
>>>>>>>>>>>> >> >>>>> method like<br>
>>>>>>>>>>>> >> >>>>> > MatFactorNumeric) has the <br>
>>>>>>>>>>>> percent of work<br>
>>>>>>>>>>>> in the GPU.<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > Junchao: *This* implies that we <br>
>>>>>>>>>>>> have a<br>
>>>>>>>>>>>> cuSparse LU<br>
>>>>>>>>>>>> >> >>>>> factorization. Is<br>
>>>>>>>>>>>> >> >>>>> > that correct? (I don't think we do)<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > No, we don't have cuSparse LU <br>
>>>>>>>>>>>> factorization. If you check<br>
>>>>>>>>>>>> >> >>>>> > <br>
>>>>>>>>>>>> MatLUFactorSymbolic_SeqAIJCUSPARSE(),you will<br>
>>>>>>>>>>>> find it<br>
>>>>>>>>>>>> >> calls<br>
>>>>>>>>>>>> >> >>>>> > MatLUFactorSymbolic_SeqAIJ() instead.<br>
>>>>>>>>>>>> >> >>>>> > So I don't understand Chang's idea. <br>
>>>>>>>>>>>> Do you want to<br>
>>>>>>>>>>>> >> make bigger<br>
>>>>>>>>>>>> >> >>>>> blocks?<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > I think this one do both <br>
>>>>>>>>>>>> factorization and<br>
>>>>>>>>>>>> >> solve on gpu.<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > You can check the<br>
>>>>>>>>>>>> runex72_aijcusparse.sh file<br>
>>>>>>>>>>>> >> in petsc<br>
>>>>>>>>>>>> >> >>>>> install<br>
>>>>>>>>>>>> >> >>>>> > directory, and try it your <br>
>>>>>>>>>>>> self (this<br>
>>>>>>>>>>>> is only lu<br>
>>>>>>>>>>>> >> >>>>> factorization<br>
>>>>>>>>>>>> >> >>>>> > without<br>
>>>>>>>>>>>> >> >>>>> > iterative solve).<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > Chang<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > On 10/12/21 1:17 PM, Mark <br>
>>>>>>>>>>>> Adams wrote:<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > On Tue, Oct 12, 2021 at <br>
>>>>>>>>>>>> 11:19 AM<br>
>>>>>>>>>>>> Chang Liu<br>
>>>>>>>>>>>> >> >>>>> <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>>>> wrote:<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > Hi Junchao,<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > No I only needs it <br>
>>>>>>>>>>>> to be transferred<br>
>>>>>>>>>>>> >> within a<br>
>>>>>>>>>>>> >> >>>>> node. I use<br>
>>>>>>>>>>>> >> >>>>> > block-Jacobi<br>
>>>>>>>>>>>> >> >>>>> > > method and GMRES to <br>
>>>>>>>>>>>> solve the sparse<br>
>>>>>>>>>>>> >> matrix, so each<br>
>>>>>>>>>>>> >> >>>>> > direct solver will<br>
>>>>>>>>>>>> >> >>>>> > > take care of a <br>
>>>>>>>>>>>> sub-block of the<br>
>>>>>>>>>>>> whole<br>
>>>>>>>>>>>> >> matrix. In this<br>
>>>>>>>>>>>> >> >>>>> > way, I can use<br>
>>>>>>>>>>>> >> >>>>> > > one<br>
>>>>>>>>>>>> >> >>>>> > > GPU to solve one <br>
>>>>>>>>>>>> sub-block, which is<br>
>>>>>>>>>>>> >> stored within<br>
>>>>>>>>>>>> >> >>>>> one node.<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > It was stated in the<br>
>>>>>>>>>>>> documentation that<br>
>>>>>>>>>>>> >> cusparse<br>
>>>>>>>>>>>> >> >>>>> solver<br>
>>>>>>>>>>>> >> >>>>> > is slow.<br>
>>>>>>>>>>>> >> >>>>> > > However, in my test <br>
>>>>>>>>>>>> using<br>
>>>>>>>>>>>> ex72.c, the<br>
>>>>>>>>>>>> >> cusparse<br>
>>>>>>>>>>>> >> >>>>> solver is<br>
>>>>>>>>>>>> >> >>>>> > faster than<br>
>>>>>>>>>>>> >> >>>>> > > mumps or <br>
>>>>>>>>>>>> superlu_dist on CPUs.<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > Are we talking about the<br>
>>>>>>>>>>>> factorization, the<br>
>>>>>>>>>>>> >> solve, or<br>
>>>>>>>>>>>> >> >>>>> both?<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > We do not have an <br>
>>>>>>>>>>>> interface to<br>
>>>>>>>>>>>> cuSparse's LU<br>
>>>>>>>>>>>> >> >>>>> factorization (I<br>
>>>>>>>>>>>> >> >>>>> > just<br>
>>>>>>>>>>>> >> >>>>> > > learned that it exists a <br>
>>>>>>>>>>>> few weeks ago).<br>
>>>>>>>>>>>> >> >>>>> > > Perhaps your fast <br>
>>>>>>>>>>>> "cusparse solver" is<br>
>>>>>>>>>>>> >> '-pc_type lu<br>
>>>>>>>>>>>> >> >>>>> -mat_type<br>
>>>>>>>>>>>> >> >>>>> > > aijcusparse' ? This <br>
>>>>>>>>>>>> would be the CPU<br>
>>>>>>>>>>>> >> factorization,<br>
>>>>>>>>>>>> >> >>>>> which is the<br>
>>>>>>>>>>>> >> >>>>> > > dominant cost.<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > Chang<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > On 10/12/21 10:24 <br>
>>>>>>>>>>>> AM, Junchao<br>
>>>>>>>>>>>> Zhang wrote:<br>
>>>>>>>>>>>> >> >>>>> > > > Hi, Chang,<br>
>>>>>>>>>>>> >> >>>>> > > > For the mumps <br>
>>>>>>>>>>>> solver, we<br>
>>>>>>>>>>>> usually<br>
>>>>>>>>>>>> >> transfers<br>
>>>>>>>>>>>> >> >>>>> matrix<br>
>>>>>>>>>>>> >> >>>>> > and vector<br>
>>>>>>>>>>>> >> >>>>> > > data<br>
>>>>>>>>>>>> >> >>>>> > > > within a compute <br>
>>>>>>>>>>>> node. For<br>
>>>>>>>>>>>> the idea you<br>
>>>>>>>>>>>> >> >>>>> propose, it<br>
>>>>>>>>>>>> >> >>>>> > looks like<br>
>>>>>>>>>>>> >> >>>>> > > we need<br>
>>>>>>>>>>>> >> >>>>> > > > to gather data within<br>
>>>>>>>>>>>> >> MPI_COMM_WORLD, right?<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > Mark, I <br>
>>>>>>>>>>>> remember you said<br>
>>>>>>>>>>>> >> cusparse solve is<br>
>>>>>>>>>>>> >> >>>>> slow<br>
>>>>>>>>>>>> >> >>>>> > and you would<br>
>>>>>>>>>>>> >> >>>>> > > > rather do it on <br>
>>>>>>>>>>>> CPU. Is it right?<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > --Junchao Zhang<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > On Mon, Oct 11, <br>
>>>>>>>>>>>> 2021 at 10:25 PM<br>
>>>>>>>>>>>> >> Chang Liu via<br>
>>>>>>>>>>>> >> >>>>> petsc-users<br>
>>>>>>>>>>>> >> >>>>> > > > <br>
>>>>>>>>>>>> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > wrote:<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > Hi,<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > Currently, it <br>
>>>>>>>>>>>> is possible<br>
>>>>>>>>>>>> to use<br>
>>>>>>>>>>>> >> mumps<br>
>>>>>>>>>>>> >> >>>>> solver in<br>
>>>>>>>>>>>> >> >>>>> > PETSC with<br>
>>>>>>>>>>>> >> >>>>> > > > <br>
>>>>>>>>>>>> -mat_mumps_use_omp_threads<br>
>>>>>>>>>>>> >> option, so that<br>
>>>>>>>>>>>> >> >>>>> > multiple MPI<br>
>>>>>>>>>>>> >> >>>>> > > processes will<br>
>>>>>>>>>>>> >> >>>>> > > > transfer the <br>
>>>>>>>>>>>> matrix and<br>
>>>>>>>>>>>> rhs data<br>
>>>>>>>>>>>> >> to the master<br>
>>>>>>>>>>>> >> >>>>> > rank, and then<br>
>>>>>>>>>>>> >> >>>>> > > master<br>
>>>>>>>>>>>> >> >>>>> > > > rank will <br>
>>>>>>>>>>>> call mumps with<br>
>>>>>>>>>>>> OpenMP<br>
>>>>>>>>>>>> >> to solve<br>
>>>>>>>>>>>> >> >>>>> the matrix.<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > I wonder if <br>
>>>>>>>>>>>> someone can<br>
>>>>>>>>>>>> develop<br>
>>>>>>>>>>>> >> similar<br>
>>>>>>>>>>>> >> >>>>> option for<br>
>>>>>>>>>>>> >> >>>>> > cusparse<br>
>>>>>>>>>>>> >> >>>>> > > solver.<br>
>>>>>>>>>>>> >> >>>>> > > > Right now, <br>
>>>>>>>>>>>> this solver<br>
>>>>>>>>>>>> does not<br>
>>>>>>>>>>>> >> work with<br>
>>>>>>>>>>>> >> >>>>> > mpiaijcusparse. I<br>
>>>>>>>>>>>> >> >>>>> > > think a<br>
>>>>>>>>>>>> >> >>>>> > > > possible <br>
>>>>>>>>>>>> workaround is to<br>
>>>>>>>>>>>> >> transfer all the<br>
>>>>>>>>>>>> >> >>>>> matrix<br>
>>>>>>>>>>>> >> >>>>> > data to one MPI<br>
>>>>>>>>>>>> >> >>>>> > > > process, and <br>
>>>>>>>>>>>> then upload the<br>
>>>>>>>>>>>> >> data to GPU to<br>
>>>>>>>>>>>> >> >>>>> solve.<br>
>>>>>>>>>>>> >> >>>>> > In this<br>
>>>>>>>>>>>> >> >>>>> > > way, one can<br>
>>>>>>>>>>>> >> >>>>> > > > use cusparse <br>
>>>>>>>>>>>> solver for a MPI<br>
>>>>>>>>>>>> >> program.<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > > > Chang<br>
>>>>>>>>>>>> >> >>>>> > > > --<br>
>>>>>>>>>>>> >> >>>>> > > > Chang Liu<br>
>>>>>>>>>>>> >> >>>>> > > > Staff <br>
>>>>>>>>>>>> Research Physicist<br>
>>>>>>>>>>>> >> >>>>> > > > +1 609 243 3438<br>
>>>>>>>>>>>> >> >>>>> > > > <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > > Princeton <br>
>>>>>>>>>>>> Plasma Physics<br>
>>>>>>>>>>>> Laboratory<br>
>>>>>>>>>>>> >> >>>>> > > > 100 <br>
>>>>>>>>>>>> Stellarator Rd,<br>
>>>>>>>>>>>> Princeton NJ<br>
>>>>>>>>>>>> >> 08540, USA<br>
>>>>>>>>>>>> >> >>>>> > > ><br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> > > --<br>
>>>>>>>>>>>> >> >>>>> > > Chang Liu<br>
>>>>>>>>>>>> >> >>>>> > > Staff Research Physicist<br>
>>>>>>>>>>>> >> >>>>> > > +1 609 243 3438<br>
>>>>>>>>>>>> >> >>>>> > > <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> > <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>>><br>
>>>>>>>>>>>> >> >>>>> > > Princeton Plasma <br>
>>>>>>>>>>>> Physics Laboratory<br>
>>>>>>>>>>>> >> >>>>> > > 100 Stellarator Rd, <br>
>>>>>>>>>>>> Princeton NJ<br>
>>>>>>>>>>>> 08540, USA<br>
>>>>>>>>>>>> >> >>>>> > ><br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> > --<br>
>>>>>>>>>>>> >> >>>>> > Chang Liu<br>
>>>>>>>>>>>> >> >>>>> > Staff Research Physicist<br>
>>>>>>>>>>>> >> >>>>> > +1 609 243 3438<br>
>>>>>>>>>>>> >> >>>>> > <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>>><br>
>>>>>>>>>>>> >> >>>>> > Princeton Plasma Physics <br>
>>>>>>>>>>>> Laboratory<br>
>>>>>>>>>>>> >> >>>>> > 100 Stellarator Rd, <br>
>>>>>>>>>>>> Princeton NJ 08540, USA<br>
>>>>>>>>>>>> >> >>>>> ><br>
>>>>>>>>>>>> >> >>>>> -- Chang Liu<br>
>>>>>>>>>>>> >> >>>>> Staff Research Physicist<br>
>>>>>>>>>>>> >> >>>>> +1 609 243 3438<br>
>>>>>>>>>>>> >> >>>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> >> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>>><br>
>>>>>>>>>>>> >> >>>>> Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>>> >> >>>>> 100 Stellarator Rd, Princeton NJ <br>
>>>>>>>>>>>> 08540, USA<br>
>>>>>>>>>>>> >> >>>><br>
>>>>>>>>>>>> >> >>>> --<br>
>>>>>>>>>>>> >> >>>> Chang Liu<br>
>>>>>>>>>>>> >> >>>> Staff Research Physicist<br>
>>>>>>>>>>>> >> >>>> +1 609 243 3438<br>
>>>>>>>>>>>> >> >>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> >>>> Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>>> >> >>>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>>>>>>> >> >><br>
>>>>>>>>>>>> >> >> --<br>
>>>>>>>>>>>> >> >> Chang Liu<br>
>>>>>>>>>>>> >> >> Staff Research Physicist<br>
>>>>>>>>>>>> >> >> +1 609 243 3438<br>
>>>>>>>>>>>> >> >> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> >> Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>>> >> >> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>>>>>>> >> ><br>
>>>>>>>>>>>> >> -- Chang Liu<br>
>>>>>>>>>>>> >> Staff Research Physicist<br>
>>>>>>>>>>>> >> +1 609 243 3438<br>
>>>>>>>>>>>> >> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>>><br>
>>>>>>>>>>>> >> Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>>> >> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>>>>>>> ><br>
>>>>>>>>>>>> > --<br>
>>>>>>>>>>>> > Chang Liu<br>
>>>>>>>>>>>> > Staff Research Physicist<br>
>>>>>>>>>>>> > +1 609 243 3438<br>
>>>>>>>>>>>> > <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <br>
>>>>>>>>>>>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
>>>>>>>>>>>> > Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>>> > 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>>>>>><br>
>>>>>>>>>>> -- <br>
>>>>>>>>>>> Chang Liu<br>
>>>>>>>>>>> Staff Research Physicist<br>
>>>>>>>>>>> +1 609 243 3438<br>
>>>>>>>>>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>>>> Princeton Plasma Physics Laboratory<br>
>>>>>>>>>>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>>>><br>
>>>>>>>>> -- <br>
>>>>>>>>> Chang Liu<br>
>>>>>>>>> Staff Research Physicist<br>
>>>>>>>>> +1 609 243 3438<br>
>>>>>>>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>>>> Princeton Plasma Physics Laboratory<br>
>>>>>>>>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>>>><br>
>>>>>>> -- <br>
>>>>>>> Chang Liu<br>
>>>>>>> Staff Research Physicist<br>
>>>>>>> +1 609 243 3438<br>
>>>>>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>>>> Princeton Plasma Physics Laboratory<br>
>>>>>>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>>><br>
>>>>> -- <br>
>>>>> Chang Liu<br>
>>>>> Staff Research Physicist<br>
>>>>> +1 609 243 3438<br>
>>>>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>>>>> Princeton Plasma Physics Laboratory<br>
>>>>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
>>>><br>
>><br>
>> -- <br>
>> Chang Liu<br>
>> Staff Research Physicist<br>
>> +1 609 243 3438<br>
>> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
>> Princeton Plasma Physics Laboratory<br>
>> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
> <br>
<br>
-- <br>
Chang Liu<br>
Staff Research Physicist<br>
+1 609 243 3438<br>
<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a><br>
Princeton Plasma Physics Laboratory<br>
100 Stellarator Rd, Princeton NJ 08540, USA<br>
</blockquote></div>
</blockquote></div>