From eda.oktay at metu.edu.tr Mon Jul 1 03:00:42 2019 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 1 Jul 2019 11:00:42 +0300 Subject: [petsc-users] Imbalance in ParMETIS In-Reply-To: References: Message-ID: Dear Matt, In order to get a nice cut, I am trying to change the balance, that is why I tried to change ubvec. But still I couldn't understand why it didn't change anything. I tried to change vertex weights but it only made Max and Min change but it didn't change the cut. Thanks, Eda On Fri, Jun 28, 2019, 1:49 PM Matthew Knepley wrote: > On Fri, Jun 28, 2019 at 5:38 AM Eda Oktay via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello everyone, >> >> I am trying to change load balance in ParMETIS in order to give different >> number of elements to different processors. In the manual, it states that >> changing "ubvec" parameter should work. So, instead of 1.05, I gave 1.2,1.5 >> and so on but load balance didn't change. I always get Max 5 Min 5 when I >> view load balance. The partitioning part of my code is: >> > > ParMetis is always balancing. ubvec changes the balance between having > even partitions and a nice cut. If > you want unbalanced partitions, you have to change the vertex weights. > > Thanks, > > Matt > > >> ierr = MatConvert(A,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr); >> >> ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr); >> ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr); >> >> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr); >> ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr); >> MatPartitioningViewImbalance(part,partitioning); >> >> What should I do? Why changing ubvec didn't change anything, am I >> missing something? >> >> Thanks, >> >> Eda >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amfoggia at gmail.com Mon Jul 1 04:10:03 2019 From: amfoggia at gmail.com (Ale Foggia) Date: Mon, 1 Jul 2019 11:10:03 +0200 Subject: [petsc-users] Communication during MatAssemblyEnd In-Reply-To: References: <845985ccc231460caea935e3e360c87c@DM6PR09MB3369.namprd09.prod.outlook.com> Message-ID: Oh, I also got the same error when I switched to the newest version of SLEPc (using OpenBlas), and I don't know where it is coming from. Can you tell me which version of SLEPc and PETSc are you using? And, are you using MKL? Thanks for trying :) El vie., 28 jun. 2019 a las 16:57, Zhang, Junchao () escribi?: > Ran with 64 nodes and 32 ranks/node, met slepc errors and did not know > how to proceed :( > > [363]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [363]PETSC ERROR: Error in external library > [363]PETSC ERROR: Error in LAPACK subroutine steqr: info=0 > [363]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [363]PETSC ERROR: Petsc Development GIT revision: v3.11.2-1052-gf1480a5c > GIT Date: 2019-06-22 21:39:54 +0000 > [363]PETSC ERROR: /tmp/main.x on a arch-cray-xc40-knl-opt named nid03387 > by jczhang Fri Jun 28 07:26:59 2019 > [1225]PETSC ERROR: #2 DSSolve() line 586 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [1225]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [1225]PETSC ERROR: #4 EPSSolve() line 149 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > [240]PETSC ERROR: #2 DSSolve() line 586 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [240]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [240]PETSC ERROR: #4 EPSSolve() line 149 in > /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > > --Junchao Zhang > > > On Fri, Jun 28, 2019 at 4:02 AM Ale Foggia wrote: > >> Junchao, >> I'm sorry for the late response. >> >> El mi?., 26 jun. 2019 a las 16:39, Zhang, Junchao () >> escribi?: >> >>> Ale, >>> The job got a chance to run but failed with out-of-memory, "Some of your >>> processes may have been killed by the cgroup out-of-memory handler." >>> >> >> I mentioned that I used 1024 nodes and 32 processes on each node because >> the application needs a lot of memory. I think that for a system of size >> 38, one needs above 256 nodes for sure (assuming only 32 procs per node). I >> would try with 512 if it's possible. >> >> I also tried with 128 core with ./main.x 2 ... and got a weird error >>> message "The size of the basis has to be at least equal to the number >>> of MPI processes used." >>> >> >> The error comes from the fact that you put a system size of only 2 which >> is too small. >> I can also see the problem in the assembly with system sizes smaller than >> 38, so you can try with like 30 (for which I also have a log). In that case >> I run with 64 nodes and 32 processes per node. I think the problem may also >> fit in 32 nodes. >> >> --Junchao Zhang >>> >>> >>> On Tue, Jun 25, 2019 at 11:24 PM Junchao Zhang >>> wrote: >>> >>>> Ale, >>>> I successfully built your code and submitted a job to the NERSC Cori >>>> machine requiring 32768 KNL cores and one and a half hours. It is estimated >>>> to run in 3 days. If you also observed the same problem with less cores, >>>> what is your input arguments? Currently, I use what in your log file, >>>> ./main.x 38 -nn -j1 1.0 -d1 1.0 -eps_type krylovschur -eps_tol 1e-9 >>>> -log_view >>>> The smaller the better. Thanks. >>>> --Junchao Zhang >>>> >>>> >>>> On Mon, Jun 24, 2019 at 6:20 AM Ale Foggia wrote: >>>> >>>>> Yes, I used KNL nodes. I you can perform the test would be great. >>>>> Could it be that I'm not using the correct configuration of the KNL nodes? >>>>> These are the environment variables I set: >>>>> MKL_NUM_THREADS=1 >>>>> OMP_NUM_THREADS=1 >>>>> KMP_HW_SUBSET=1t >>>>> KMP_AFFINITY=compact >>>>> I_MPI_PIN_DOMAIN=socket >>>>> I_MPI_PIN_PROCESSOR_LIST=0-63 >>>>> MKL_DYNAMIC=0 >>>>> >>>>> The code is in https://github.com/amfoggia/LSQuantumED and it has a >>>>> readme to compile it and run it. When I run the test I used only 32 >>>>> processors per node, and I used 1024 nodes in total, and it's for nspins=38. >>>>> Thank you >>>>> >>>>> El vie., 21 jun. 2019 a las 20:03, Zhang, Junchao (< >>>>> jczhang at mcs.anl.gov>) escribi?: >>>>> >>>>>> Ale, >>>>>> Did you use Intel KNL nodes? Mr. Hong (cc'ed) did experiments on >>>>>> KNL nodes one year ago. He used 32768 processors and called MatAssemblyEnd >>>>>> 118 times and it used only 1.5 seconds in total. So I guess something was >>>>>> wrong with your test. If you can share your code, I can have a test on our >>>>>> machine to see how it goes. >>>>>> Thanks. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jun 21, 2019 at 11:00 AM Junchao Zhang >>>>>> wrote: >>>>>> >>>>>>> MatAssembly was called once (in stage 5) and cost 2.5% of the total >>>>>>> time. Look at stage 5. It says MatAssemblyBegin calls BuildTwoSidedF, >>>>>>> which does global synchronization. The high max/min ratio means load >>>>>>> imbalance. What I do not understand is MatAssemblyEnd. The ratio is 1.0. It >>>>>>> means processors are already synchronized. With 32768 processors, there are >>>>>>> 1.2e+06 messages with average length 1.9e+06 bytes. So each processor sends >>>>>>> 36 (1.2e+06/32768) ~2MB messages and it takes 54 seconds. Another chance is >>>>>>> the reduction at MatAssemblyEnd. I don't know why it needs 8 reductions. >>>>>>> In my mind, one is enough. I need to look at the code. >>>>>>> >>>>>>> Summary of Stages: ----- Time ------ ----- Flop ------ --- >>>>>>> Messages --- -- Message Lengths -- -- Reductions -- >>>>>>> Avg %Total Avg %Total Count >>>>>>> %Total Avg %Total Count %Total >>>>>>> 0: Main Stage: 8.5045e+02 13.0% 3.0633e+15 14.0% 8.196e+07 >>>>>>> 13.1% 7.768e+06 13.1% 2.530e+02 13.0% >>>>>>> 1: Create Basis: 7.9234e-02 0.0% 0.0000e+00 0.0% 0.000e+00 >>>>>>> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>>>>>> 2: Create Lattice: 8.3944e-05 0.0% 0.0000e+00 0.0% 0.000e+00 >>>>>>> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>>>>>> 3: Create Hamilt: 1.0694e+02 1.6% 0.0000e+00 0.0% 0.000e+00 >>>>>>> 0.0% 0.000e+00 0.0% 2.000e+00 0.1% >>>>>>> 5: Offdiag: 1.6525e+02 2.5% 0.0000e+00 0.0% 1.188e+06 >>>>>>> 0.2% 1.942e+06 0.0% 8.000e+00 0.4% >>>>>>> 6: Phys quantities: 5.4045e+03 82.8% 1.8866e+16 86.0% 5.417e+08 >>>>>>> 86.7% 7.768e+06 86.8% 1.674e+03 86.1% >>>>>>> >>>>>>> --- Event Stage 5: Offdiag >>>>>>> BuildTwoSidedF 1 1.0 7.1565e+01 148448.9 0.00e+00 0.0 >>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >>>>>>> MatAssemblyBegin 1 1.0 7.1565e+01 127783.7 0.00e+00 0.0 >>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >>>>>>> MatAssemblyEnd 1 1.0 5.3762e+01 1.0 0.00e+00 0.0 >>>>>>> 1.2e+06 1.9e+06 8.0e+00 1 0 0 0 0 33 0100100100 0 >>>>>>> VecSet 1 1.0 7.5533e-02 9.0 0.00e+00 0.0 >>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 21, 2019 at 10:34 AM Smith, Barry F. >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> The load balance is definitely out of whack. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> BuildTwoSidedF 1 1.0 1.6722e-0241.0 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatMult 138 1.0 2.6604e+02 7.4 3.19e+10 2.1 8.2e+07 >>>>>>>> 7.8e+06 0.0e+00 2 4 13 13 0 15 25100100 0 2935476 >>>>>>>> MatAssemblyBegin 1 1.0 1.6807e-0236.1 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyEnd 1 1.0 3.5680e-01 3.9 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecNorm 2 1.0 4.4252e+0174.8 1.73e+07 1.0 0.0e+00 >>>>>>>> 0.0e+00 2.0e+00 1 0 0 0 0 5 0 0 0 1 12780 >>>>>>>> VecCopy 6 1.0 6.5655e-02 2.6 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecAXPY 2 1.0 1.3793e-02 2.7 1.73e+07 1.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41000838 >>>>>>>> VecScatterBegin 138 1.0 1.1653e+0285.8 0.00e+00 0.0 8.2e+07 >>>>>>>> 7.8e+06 0.0e+00 1 0 13 13 0 4 0100100 0 0 >>>>>>>> VecScatterEnd 138 1.0 1.3653e+0222.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 >>>>>>>> VecSetRandom 1 1.0 9.6668e-01 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> >>>>>>>> Note that VecCopy/AXPY/SetRandom which are all embarrassingly >>>>>>>> parallel have a balance ratio above 2 which means some processes have more >>>>>>>> than twice the work of others. Meanwhile the ratio for anything with >>>>>>>> communication is extremely in balanced, some processes get to the >>>>>>>> synchronization point well before other processes. >>>>>>>> >>>>>>>> The first thing I would do is worry about the load imbalance, what >>>>>>>> is its cause? is it one process with much less work than others (not great >>>>>>>> but not terrible) or is it one process with much more work then the others >>>>>>>> (terrible) or something in between. I think once you get a handle on the >>>>>>>> load balance the rest may fall into place, otherwise we still have some >>>>>>>> exploring to do. This is not expected behavior for a good machine with a >>>>>>>> good network and a well balanced job. After you understand the load >>>>>>>> balancing you may need to use one of the parallel performance visualization >>>>>>>> tools to see why the synchronization is out of whack. >>>>>>>> >>>>>>>> Good luck >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> > On Jun 21, 2019, at 9:27 AM, Ale Foggia >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > I'm sending one with a bit less time. >>>>>>>> > I'm timing the functions also with std::chronos and for the case >>>>>>>> of 180 seconds the program runs out of memory (and crushes) before the >>>>>>>> PETSc log gets to be printed, so I know the time only from my function. >>>>>>>> Anyway, in every case, the times between std::chronos and the PETSc log >>>>>>>> match. >>>>>>>> > >>>>>>>> > (The large times are in part "4b- Building offdiagonal part" or >>>>>>>> "Event Stage 5: Offdiag"). >>>>>>>> > >>>>>>>> > El vie., 21 jun. 2019 a las 16:09, Zhang, Junchao (< >>>>>>>> jczhang at mcs.anl.gov>) escribi?: >>>>>>>> > >>>>>>>> > >>>>>>>> > On Fri, Jun 21, 2019 at 8:07 AM Ale Foggia >>>>>>>> wrote: >>>>>>>> > Thanks both of you for your answers, >>>>>>>> > >>>>>>>> > El jue., 20 jun. 2019 a las 22:20, Smith, Barry F. (< >>>>>>>> bsmith at mcs.anl.gov>) escribi?: >>>>>>>> > >>>>>>>> > Note that this is a one time cost if the nonzero structure of >>>>>>>> the matrix stays the same. It will not happen in future MatAssemblies. >>>>>>>> > >>>>>>>> > > On Jun 20, 2019, at 3:16 PM, Zhang, Junchao via petsc-users < >>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>> > > >>>>>>>> > > Those messages were used to build MatMult communication pattern >>>>>>>> for the matrix. They were not part of the matrix entries-passing you >>>>>>>> imagined, but indeed happened in MatAssemblyEnd. If you want to make sure >>>>>>>> processors do not set remote entries, you can use >>>>>>>> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE), which will generate an >>>>>>>> error when an off-proc entry is set. >>>>>>>> > >>>>>>>> > I started being concerned about this when I saw that the assembly >>>>>>>> was taking a few hundreds of seconds in my code, like 180 seconds, which >>>>>>>> for me it's a considerable time. Do you think (or maybe you need more >>>>>>>> information to answer this) that this time is "reasonable" for >>>>>>>> communicating the pattern for the matrix? I already checked that I'm not >>>>>>>> setting any remote entries. >>>>>>>> > It is not reasonable. Could you send log view of that test with >>>>>>>> 180 seconds MatAssembly? >>>>>>>> > >>>>>>>> > Also I see (in my code) that even if there are no messages being >>>>>>>> passed during MatAssemblyBegin, it is taking time and the "ratio" is very >>>>>>>> big. >>>>>>>> > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > --Junchao Zhang >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > On Thu, Jun 20, 2019 at 4:13 AM Ale Foggia via petsc-users < >>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>> > > Hello all! >>>>>>>> > > >>>>>>>> > > During the conference I showed you a problem happening during >>>>>>>> MatAssemblyEnd in a particular code that I have. Now, I tried the same with >>>>>>>> a simple code (a symmetric problem corresponding to the Laplacian operator >>>>>>>> in 1D, from the SLEPc Hands-On exercises). As I understand (and please, >>>>>>>> correct me if I'm wrong), in this case the elements of the matrix are >>>>>>>> computed locally by each process so there should not be any communication >>>>>>>> during the assembly. However, in the log I get that there are messages >>>>>>>> being passed. Also, the number of messages changes with the number of >>>>>>>> processes used and the size of the matrix. Could you please help me >>>>>>>> understand this? >>>>>>>> > > >>>>>>>> > > I attach the code I used and the log I get for a small problem. >>>>>>>> > > >>>>>>>> > > Cheers, >>>>>>>> > > Ale >>>>>>>> > > >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Jul 1 04:28:04 2019 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 1 Jul 2019 11:28:04 +0200 Subject: [petsc-users] Communication during MatAssemblyEnd In-Reply-To: References: <845985ccc231460caea935e3e360c87c@DM6PR09MB3369.namprd09.prod.outlook.com> Message-ID: You can try the following: - Try with a different DS method: -ds_method 1 or -ds_method 2 (see DSSetMethod) - Run with -ds_parallel synchronized (see DSSetParallel) If it does not help, send a reproducible code to slepc-maint Jose > El 1 jul 2019, a las 11:10, Ale Foggia via petsc-users escribi?: > > Oh, I also got the same error when I switched to the newest version of SLEPc (using OpenBlas), and I don't know where it is coming from. > Can you tell me which version of SLEPc and PETSc are you using? And, are you using MKL? > Thanks for trying :) > > El vie., 28 jun. 2019 a las 16:57, Zhang, Junchao () escribi?: > Ran with 64 nodes and 32 ranks/node, met slepc errors and did not know how to proceed :( > > [363]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [363]PETSC ERROR: Error in external library > [363]PETSC ERROR: Error in LAPACK subroutine steqr: info=0 > [363]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [363]PETSC ERROR: Petsc Development GIT revision: v3.11.2-1052-gf1480a5c GIT Date: 2019-06-22 21:39:54 +0000 > [363]PETSC ERROR: /tmp/main.x on a arch-cray-xc40-knl-opt named nid03387 by jczhang Fri Jun 28 07:26:59 2019 > [1225]PETSC ERROR: #2 DSSolve() line 586 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [1225]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [1225]PETSC ERROR: #4 EPSSolve() line 149 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > [240]PETSC ERROR: #2 DSSolve() line 586 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [240]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [240]PETSC ERROR: #4 EPSSolve() line 149 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > > --Junchao Zhang > > > On Fri, Jun 28, 2019 at 4:02 AM Ale Foggia wrote: > Junchao, > I'm sorry for the late response. > > El mi?., 26 jun. 2019 a las 16:39, Zhang, Junchao () escribi?: > Ale, > The job got a chance to run but failed with out-of-memory, "Some of your processes may have been killed by the cgroup out-of-memory handler." > > I mentioned that I used 1024 nodes and 32 processes on each node because the application needs a lot of memory. I think that for a system of size 38, one needs above 256 nodes for sure (assuming only 32 procs per node). I would try with 512 if it's possible. > > I also tried with 128 core with ./main.x 2 ... and got a weird error message "The size of the basis has to be at least equal to the number of MPI processes used." > > The error comes from the fact that you put a system size of only 2 which is too small. > I can also see the problem in the assembly with system sizes smaller than 38, so you can try with like 30 (for which I also have a log). In that case I run with 64 nodes and 32 processes per node. I think the problem may also fit in 32 nodes. > > --Junchao Zhang > > > On Tue, Jun 25, 2019 at 11:24 PM Junchao Zhang wrote: > Ale, > I successfully built your code and submitted a job to the NERSC Cori machine requiring 32768 KNL cores and one and a half hours. It is estimated to run in 3 days. If you also observed the same problem with less cores, what is your input arguments? Currently, I use what in your log file, ./main.x 38 -nn -j1 1.0 -d1 1.0 -eps_type krylovschur -eps_tol 1e-9 -log_view > The smaller the better. Thanks. > --Junchao Zhang > > > On Mon, Jun 24, 2019 at 6:20 AM Ale Foggia wrote: > Yes, I used KNL nodes. I you can perform the test would be great. Could it be that I'm not using the correct configuration of the KNL nodes? These are the environment variables I set: > MKL_NUM_THREADS=1 > OMP_NUM_THREADS=1 > KMP_HW_SUBSET=1t > KMP_AFFINITY=compact > I_MPI_PIN_DOMAIN=socket > I_MPI_PIN_PROCESSOR_LIST=0-63 > MKL_DYNAMIC=0 > > The code is in https://github.com/amfoggia/LSQuantumED and it has a readme to compile it and run it. When I run the test I used only 32 processors per node, and I used 1024 nodes in total, and it's for nspins=38. > Thank you > > El vie., 21 jun. 2019 a las 20:03, Zhang, Junchao () escribi?: > Ale, > Did you use Intel KNL nodes? Mr. Hong (cc'ed) did experiments on KNL nodes one year ago. He used 32768 processors and called MatAssemblyEnd 118 times and it used only 1.5 seconds in total. So I guess something was wrong with your test. If you can share your code, I can have a test on our machine to see how it goes. > Thanks. > --Junchao Zhang > > > On Fri, Jun 21, 2019 at 11:00 AM Junchao Zhang wrote: > MatAssembly was called once (in stage 5) and cost 2.5% of the total time. Look at stage 5. It says MatAssemblyBegin calls BuildTwoSidedF, which does global synchronization. The high max/min ratio means load imbalance. What I do not understand is MatAssemblyEnd. The ratio is 1.0. It means processors are already synchronized. With 32768 processors, there are 1.2e+06 messages with average length 1.9e+06 bytes. So each processor sends 36 (1.2e+06/32768) ~2MB messages and it takes 54 seconds. Another chance is the reduction at MatAssemblyEnd. I don't know why it needs 8 reductions. In my mind, one is enough. I need to look at the code. > > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total Count %Total Avg %Total Count %Total > 0: Main Stage: 8.5045e+02 13.0% 3.0633e+15 14.0% 8.196e+07 13.1% 7.768e+06 13.1% 2.530e+02 13.0% > 1: Create Basis: 7.9234e-02 0.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 2: Create Lattice: 8.3944e-05 0.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 3: Create Hamilt: 1.0694e+02 1.6% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+00 0.1% > 5: Offdiag: 1.6525e+02 2.5% 0.0000e+00 0.0% 1.188e+06 0.2% 1.942e+06 0.0% 8.000e+00 0.4% > 6: Phys quantities: 5.4045e+03 82.8% 1.8866e+16 86.0% 5.417e+08 86.7% 7.768e+06 86.8% 1.674e+03 86.1% > > --- Event Stage 5: Offdiag > BuildTwoSidedF 1 1.0 7.1565e+01 148448.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 > MatAssemblyBegin 1 1.0 7.1565e+01 127783.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 > MatAssemblyEnd 1 1.0 5.3762e+01 1.0 0.00e+00 0.0 1.2e+06 1.9e+06 8.0e+00 1 0 0 0 0 33 0100100100 0 > VecSet 1 1.0 7.5533e-02 9.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > --Junchao Zhang > > > On Fri, Jun 21, 2019 at 10:34 AM Smith, Barry F. wrote: > > The load balance is definitely out of whack. > > > > BuildTwoSidedF 1 1.0 1.6722e-0241.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 138 1.0 2.6604e+02 7.4 3.19e+10 2.1 8.2e+07 7.8e+06 0.0e+00 2 4 13 13 0 15 25100100 0 2935476 > MatAssemblyBegin 1 1.0 1.6807e-0236.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 1 1.0 3.5680e-01 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNorm 2 1.0 4.4252e+0174.8 1.73e+07 1.0 0.0e+00 0.0e+00 2.0e+00 1 0 0 0 0 5 0 0 0 1 12780 > VecCopy 6 1.0 6.5655e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 2 1.0 1.3793e-02 2.7 1.73e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41000838 > VecScatterBegin 138 1.0 1.1653e+0285.8 0.00e+00 0.0 8.2e+07 7.8e+06 0.0e+00 1 0 13 13 0 4 0100100 0 0 > VecScatterEnd 138 1.0 1.3653e+0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 > VecSetRandom 1 1.0 9.6668e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > Note that VecCopy/AXPY/SetRandom which are all embarrassingly parallel have a balance ratio above 2 which means some processes have more than twice the work of others. Meanwhile the ratio for anything with communication is extremely in balanced, some processes get to the synchronization point well before other processes. > > The first thing I would do is worry about the load imbalance, what is its cause? is it one process with much less work than others (not great but not terrible) or is it one process with much more work then the others (terrible) or something in between. I think once you get a handle on the load balance the rest may fall into place, otherwise we still have some exploring to do. This is not expected behavior for a good machine with a good network and a well balanced job. After you understand the load balancing you may need to use one of the parallel performance visualization tools to see why the synchronization is out of whack. > > Good luck > > Barry > > > > On Jun 21, 2019, at 9:27 AM, Ale Foggia wrote: > > > > I'm sending one with a bit less time. > > I'm timing the functions also with std::chronos and for the case of 180 seconds the program runs out of memory (and crushes) before the PETSc log gets to be printed, so I know the time only from my function. Anyway, in every case, the times between std::chronos and the PETSc log match. > > > > (The large times are in part "4b- Building offdiagonal part" or "Event Stage 5: Offdiag"). > > > > El vie., 21 jun. 2019 a las 16:09, Zhang, Junchao () escribi?: > > > > > > On Fri, Jun 21, 2019 at 8:07 AM Ale Foggia wrote: > > Thanks both of you for your answers, > > > > El jue., 20 jun. 2019 a las 22:20, Smith, Barry F. () escribi?: > > > > Note that this is a one time cost if the nonzero structure of the matrix stays the same. It will not happen in future MatAssemblies. > > > > > On Jun 20, 2019, at 3:16 PM, Zhang, Junchao via petsc-users wrote: > > > > > > Those messages were used to build MatMult communication pattern for the matrix. They were not part of the matrix entries-passing you imagined, but indeed happened in MatAssemblyEnd. If you want to make sure processors do not set remote entries, you can use MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE), which will generate an error when an off-proc entry is set. > > > > I started being concerned about this when I saw that the assembly was taking a few hundreds of seconds in my code, like 180 seconds, which for me it's a considerable time. Do you think (or maybe you need more information to answer this) that this time is "reasonable" for communicating the pattern for the matrix? I already checked that I'm not setting any remote entries. > > It is not reasonable. Could you send log view of that test with 180 seconds MatAssembly? > > > > Also I see (in my code) that even if there are no messages being passed during MatAssemblyBegin, it is taking time and the "ratio" is very big. > > > > > > > > > > > --Junchao Zhang > > > > > > > > > On Thu, Jun 20, 2019 at 4:13 AM Ale Foggia via petsc-users wrote: > > > Hello all! > > > > > > During the conference I showed you a problem happening during MatAssemblyEnd in a particular code that I have. Now, I tried the same with a simple code (a symmetric problem corresponding to the Laplacian operator in 1D, from the SLEPc Hands-On exercises). As I understand (and please, correct me if I'm wrong), in this case the elements of the matrix are computed locally by each process so there should not be any communication during the assembly. However, in the log I get that there are messages being passed. Also, the number of messages changes with the number of processes used and the size of the matrix. Could you please help me understand this? > > > > > > I attach the code I used and the log I get for a small problem. > > > > > > Cheers, > > > Ale > > > > > > > > From knepley at gmail.com Mon Jul 1 05:09:56 2019 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Jul 2019 05:09:56 -0500 Subject: [petsc-users] Imbalance in ParMETIS In-Reply-To: References: Message-ID: On Mon, Jul 1, 2019 at 3:00 AM Eda Oktay wrote: > Dear Matt, > > In order to get a nice cut, I am trying to change the balance, that is why > I tried to change ubvec. But still I couldn't understand why it didn't > change anything. > > I tried to change vertex weights but it only made Max and Min change but > it didn't change the cut. > I think you are confusing concepts. The "cut" is the number of edges discarded when you divide the mesh into pieces, but each piece is supposed to have an equal size. The only way you can get the partitioner to divide the mesh into pieces of unequal size is to use vertex weights. Thanks, Matt > Thanks, > > Eda > > > On Fri, Jun 28, 2019, 1:49 PM Matthew Knepley wrote: > >> On Fri, Jun 28, 2019 at 5:38 AM Eda Oktay via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello everyone, >>> >>> I am trying to change load balance in ParMETIS in order to give >>> different number of elements to different processors. In the manual, it >>> states that changing "ubvec" parameter should work. So, instead of 1.05, I >>> gave 1.2,1.5 and so on but load balance didn't change. I always get Max 5 >>> Min 5 when I view load balance. The partitioning part of my code is: >>> >> >> ParMetis is always balancing. ubvec changes the balance between having >> even partitions and a nice cut. If >> you want unbalanced partitions, you have to change the vertex weights. >> >> Thanks, >> >> Matt >> >> >>> ierr = MatConvert(A,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr); >>> >>> ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr); >>> ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr); >>> >>> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr); >>> ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr); >>> MatPartitioningViewImbalance(part,partitioning); >>> >>> What should I do? Why changing ubvec didn't change anything, am I >>> missing something? >>> >>> Thanks, >>> >>> Eda >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Mon Jul 1 11:36:27 2019 From: jczhang at mcs.anl.gov (Zhang, Junchao) Date: Mon, 1 Jul 2019 16:36:27 +0000 Subject: [petsc-users] Communication during MatAssemblyEnd In-Reply-To: References: <845985ccc231460caea935e3e360c87c@DM6PR09MB3369.namprd09.prod.outlook.com> Message-ID: Jose & Ale, -ds_method 2 fixed the problem. I used PETSc master (f1480a5c) and slepc master(675b89d7) through --download-slepc. I used MKL /opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/ I got the following results with 2048 processors. MatAssemblyEnd looks expensive to me. I am looking into it. --- Event Stage 5: Offdiag BuildTwoSidedF 1 1.0 1.5201e+007345.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 13 0 0 0 0 0 MatAssemblyBegin 1 1.0 1.5201e+005371.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 13 0 0 0 0 0 MatAssemblyEnd 1 1.0 1.7720e+00 1.0 0.00e+00 0.0 5.7e+04 1.3e+05 8.0e+00 2 0 0 0 1 39 0100100100 0 VecSet 1 1.0 2.4695e-02 7.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --Junchao Zhang On Mon, Jul 1, 2019 at 4:28 AM Jose E. Roman > wrote: You can try the following: - Try with a different DS method: -ds_method 1 or -ds_method 2 (see DSSetMethod) - Run with -ds_parallel synchronized (see DSSetParallel) If it does not help, send a reproducible code to slepc-maint Jose > El 1 jul 2019, a las 11:10, Ale Foggia via petsc-users > escribi?: > > Oh, I also got the same error when I switched to the newest version of SLEPc (using OpenBlas), and I don't know where it is coming from. > Can you tell me which version of SLEPc and PETSc are you using? And, are you using MKL? > Thanks for trying :) > > El vie., 28 jun. 2019 a las 16:57, Zhang, Junchao (>) escribi?: > Ran with 64 nodes and 32 ranks/node, met slepc errors and did not know how to proceed :( > > [363]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [363]PETSC ERROR: Error in external library > [363]PETSC ERROR: Error in LAPACK subroutine steqr: info=0 > [363]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [363]PETSC ERROR: Petsc Development GIT revision: v3.11.2-1052-gf1480a5c GIT Date: 2019-06-22 21:39:54 +0000 > [363]PETSC ERROR: /tmp/main.x on a arch-cray-xc40-knl-opt named nid03387 by jczhang Fri Jun 28 07:26:59 2019 > [1225]PETSC ERROR: #2 DSSolve() line 586 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [1225]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [1225]PETSC ERROR: #4 EPSSolve() line 149 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > [240]PETSC ERROR: #2 DSSolve() line 586 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c > [240]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c > [240]PETSC ERROR: #4 EPSSolve() line 149 in /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c > > --Junchao Zhang > > > On Fri, Jun 28, 2019 at 4:02 AM Ale Foggia > wrote: > Junchao, > I'm sorry for the late response. > > El mi?., 26 jun. 2019 a las 16:39, Zhang, Junchao (>) escribi?: > Ale, > The job got a chance to run but failed with out-of-memory, "Some of your processes may have been killed by the cgroup out-of-memory handler." > > I mentioned that I used 1024 nodes and 32 processes on each node because the application needs a lot of memory. I think that for a system of size 38, one needs above 256 nodes for sure (assuming only 32 procs per node). I would try with 512 if it's possible. > > I also tried with 128 core with ./main.x 2 ... and got a weird error message "The size of the basis has to be at least equal to the number of MPI processes used." > > The error comes from the fact that you put a system size of only 2 which is too small. > I can also see the problem in the assembly with system sizes smaller than 38, so you can try with like 30 (for which I also have a log). In that case I run with 64 nodes and 32 processes per node. I think the problem may also fit in 32 nodes. > > --Junchao Zhang > > > On Tue, Jun 25, 2019 at 11:24 PM Junchao Zhang > wrote: > Ale, > I successfully built your code and submitted a job to the NERSC Cori machine requiring 32768 KNL cores and one and a half hours. It is estimated to run in 3 days. If you also observed the same problem with less cores, what is your input arguments? Currently, I use what in your log file, ./main.x 38 -nn -j1 1.0 -d1 1.0 -eps_type krylovschur -eps_tol 1e-9 -log_view > The smaller the better. Thanks. > --Junchao Zhang > > > On Mon, Jun 24, 2019 at 6:20 AM Ale Foggia > wrote: > Yes, I used KNL nodes. I you can perform the test would be great. Could it be that I'm not using the correct configuration of the KNL nodes? These are the environment variables I set: > MKL_NUM_THREADS=1 > OMP_NUM_THREADS=1 > KMP_HW_SUBSET=1t > KMP_AFFINITY=compact > I_MPI_PIN_DOMAIN=socket > I_MPI_PIN_PROCESSOR_LIST=0-63 > MKL_DYNAMIC=0 > > The code is in https://github.com/amfoggia/LSQuantumED and it has a readme to compile it and run it. When I run the test I used only 32 processors per node, and I used 1024 nodes in total, and it's for nspins=38. > Thank you > > El vie., 21 jun. 2019 a las 20:03, Zhang, Junchao (>) escribi?: > Ale, > Did you use Intel KNL nodes? Mr. Hong (cc'ed) did experiments on KNL nodes one year ago. He used 32768 processors and called MatAssemblyEnd 118 times and it used only 1.5 seconds in total. So I guess something was wrong with your test. If you can share your code, I can have a test on our machine to see how it goes. > Thanks. > --Junchao Zhang > > > On Fri, Jun 21, 2019 at 11:00 AM Junchao Zhang > wrote: > MatAssembly was called once (in stage 5) and cost 2.5% of the total time. Look at stage 5. It says MatAssemblyBegin calls BuildTwoSidedF, which does global synchronization. The high max/min ratio means load imbalance. What I do not understand is MatAssemblyEnd. The ratio is 1.0. It means processors are already synchronized. With 32768 processors, there are 1.2e+06 messages with average length 1.9e+06 bytes. So each processor sends 36 (1.2e+06/32768) ~2MB messages and it takes 54 seconds. Another chance is the reduction at MatAssemblyEnd. I don't know why it needs 8 reductions. In my mind, one is enough. I need to look at the code. > > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total Count %Total Avg %Total Count %Total > 0: Main Stage: 8.5045e+02 13.0% 3.0633e+15 14.0% 8.196e+07 13.1% 7.768e+06 13.1% 2.530e+02 13.0% > 1: Create Basis: 7.9234e-02 0.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 2: Create Lattice: 8.3944e-05 0.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 3: Create Hamilt: 1.0694e+02 1.6% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+00 0.1% > 5: Offdiag: 1.6525e+02 2.5% 0.0000e+00 0.0% 1.188e+06 0.2% 1.942e+06 0.0% 8.000e+00 0.4% > 6: Phys quantities: 5.4045e+03 82.8% 1.8866e+16 86.0% 5.417e+08 86.7% 7.768e+06 86.8% 1.674e+03 86.1% > > --- Event Stage 5: Offdiag > BuildTwoSidedF 1 1.0 7.1565e+01 148448.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 > MatAssemblyBegin 1 1.0 7.1565e+01 127783.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 > MatAssemblyEnd 1 1.0 5.3762e+01 1.0 0.00e+00 0.0 1.2e+06 1.9e+06 8.0e+00 1 0 0 0 0 33 0100100100 0 > VecSet 1 1.0 7.5533e-02 9.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > --Junchao Zhang > > > On Fri, Jun 21, 2019 at 10:34 AM Smith, Barry F. > wrote: > > The load balance is definitely out of whack. > > > > BuildTwoSidedF 1 1.0 1.6722e-0241.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 138 1.0 2.6604e+02 7.4 3.19e+10 2.1 8.2e+07 7.8e+06 0.0e+00 2 4 13 13 0 15 25100100 0 2935476 > MatAssemblyBegin 1 1.0 1.6807e-0236.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 1 1.0 3.5680e-01 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNorm 2 1.0 4.4252e+0174.8 1.73e+07 1.0 0.0e+00 0.0e+00 2.0e+00 1 0 0 0 0 5 0 0 0 1 12780 > VecCopy 6 1.0 6.5655e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 2 1.0 1.3793e-02 2.7 1.73e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41000838 > VecScatterBegin 138 1.0 1.1653e+0285.8 0.00e+00 0.0 8.2e+07 7.8e+06 0.0e+00 1 0 13 13 0 4 0100100 0 0 > VecScatterEnd 138 1.0 1.3653e+0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 > VecSetRandom 1 1.0 9.6668e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > Note that VecCopy/AXPY/SetRandom which are all embarrassingly parallel have a balance ratio above 2 which means some processes have more than twice the work of others. Meanwhile the ratio for anything with communication is extremely in balanced, some processes get to the synchronization point well before other processes. > > The first thing I would do is worry about the load imbalance, what is its cause? is it one process with much less work than others (not great but not terrible) or is it one process with much more work then the others (terrible) or something in between. I think once you get a handle on the load balance the rest may fall into place, otherwise we still have some exploring to do. This is not expected behavior for a good machine with a good network and a well balanced job. After you understand the load balancing you may need to use one of the parallel performance visualization tools to see why the synchronization is out of whack. > > Good luck > > Barry > > > > On Jun 21, 2019, at 9:27 AM, Ale Foggia > wrote: > > > > I'm sending one with a bit less time. > > I'm timing the functions also with std::chronos and for the case of 180 seconds the program runs out of memory (and crushes) before the PETSc log gets to be printed, so I know the time only from my function. Anyway, in every case, the times between std::chronos and the PETSc log match. > > > > (The large times are in part "4b- Building offdiagonal part" or "Event Stage 5: Offdiag"). > > > > El vie., 21 jun. 2019 a las 16:09, Zhang, Junchao (>) escribi?: > > > > > > On Fri, Jun 21, 2019 at 8:07 AM Ale Foggia > wrote: > > Thanks both of you for your answers, > > > > El jue., 20 jun. 2019 a las 22:20, Smith, Barry F. (>) escribi?: > > > > Note that this is a one time cost if the nonzero structure of the matrix stays the same. It will not happen in future MatAssemblies. > > > > > On Jun 20, 2019, at 3:16 PM, Zhang, Junchao via petsc-users > wrote: > > > > > > Those messages were used to build MatMult communication pattern for the matrix. They were not part of the matrix entries-passing you imagined, but indeed happened in MatAssemblyEnd. If you want to make sure processors do not set remote entries, you can use MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE), which will generate an error when an off-proc entry is set. > > > > I started being concerned about this when I saw that the assembly was taking a few hundreds of seconds in my code, like 180 seconds, which for me it's a considerable time. Do you think (or maybe you need more information to answer this) that this time is "reasonable" for communicating the pattern for the matrix? I already checked that I'm not setting any remote entries. > > It is not reasonable. Could you send log view of that test with 180 seconds MatAssembly? > > > > Also I see (in my code) that even if there are no messages being passed during MatAssemblyBegin, it is taking time and the "ratio" is very big. > > > > > > > > > > > --Junchao Zhang > > > > > > > > > On Thu, Jun 20, 2019 at 4:13 AM Ale Foggia via petsc-users > wrote: > > > Hello all! > > > > > > During the conference I showed you a problem happening during MatAssemblyEnd in a particular code that I have. Now, I tried the same with a simple code (a symmetric problem corresponding to the Laplacian operator in 1D, from the SLEPc Hands-On exercises). As I understand (and please, correct me if I'm wrong), in this case the elements of the matrix are computed locally by each process so there should not be any communication during the assembly. However, in the log I get that there are messages being passed. Also, the number of messages changes with the number of processes used and the size of the matrix. Could you please help me understand this? > > > > > > I attach the code I used and the log I get for a small problem. > > > > > > Cheers, > > > Ale > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iancclin at umich.edu Mon Jul 1 17:08:51 2019 From: iancclin at umich.edu (Ian C. Lin) Date: Mon, 1 Jul 2019 18:08:51 -0400 Subject: [petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE Message-ID: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> Hi, I am recently trying to do matrix multiplication for C = A*B, where A is a sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE. In matrix A, the nonzeros are not distributed evenly across the processor, meaning that if using the default setting to let each processor own similar number of rows, the number of nonzeros owned by each processor will be significantly different. So I want to use different number of local rows for each processor. In this case, does the MPIDense matrices B and C need to be in the same row-layout as A? I mean, is something like the following is doable (A owns 3 rows and B, C own 2 rows) A B C P0 o o o o | o. o o. o o o o o o | o o o o o o o o o | o. *. ----- = ---- --------------- o o o o P1 o o o o | o o o o o In this case, the entries can be evenly distributed for B and C thus more memory efficient. But I am not sure would this make communication more complicated thus slow down the overall wall time. How would you recommend to do? a) Let rows of A and B be both evenly distributed b) Let A have different rows layout, and B, C evenly distributed c) Let A have different rows layout, and B, C follow A Or maybe other better way that I did not think about. Thanks a lot for your help, Ian From rbpiccinini at gmail.com Mon Jul 1 18:19:16 2019 From: rbpiccinini at gmail.com (Rodrigo Piccinini) Date: Mon, 1 Jul 2019 20:19:16 -0300 Subject: [petsc-users] Out of memory - diffusion equation eigenvalues Message-ID: Hello, I'm trying to solve for the first ten smallest eigenvalues of a diffusion equation in a 3d heterogeneous medium (varying diffusion coefficient), but I'm getting an error message: Traceback (most recent call last): > File "eigendrake_spe10_v2.py", line 121, in > es.solve() > File "SLEPc/EPS.pyx", line 1132, in slepc4py.SLEPc.EPS.solve > petsc4py.PETSc.Error: error code 55 > [0] EPSSolve() line 136 in > /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssolve.c > [0] EPSSetUp() line 269 in > /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssetup.c > [0] STSetUp() line 271 in > /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/interface/stsolve.c > [0] STSetUp_Shift() line 119 in > /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/impls/shift/shift.c > [0] KSPSetUp() line 391 in > /tmp/pip-req-build-070lpqqd/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 932 in > /tmp/pip-req-build-070lpqqd/src/ksp/pc/interface/precon.c > [0] PCSetUp_LU() line 99 in > /tmp/pip-req-build-070lpqqd/src/ksp/pc/impls/factor/lu/lu.c > [0] MatLUFactorSymbolic() line 3111 in > /tmp/pip-req-build-070lpqqd/src/mat/interface/matrix.c > [0] MatLUFactorSymbolic_SeqAIJ() line 366 in > /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c > [0] PetscMallocA() line 390 in > /tmp/pip-req-build-070lpqqd/src/sys/memory/mal.c > [0] MatLUFactorSymbolic_SeqAIJ() line 366 in > /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c > [0] Out of memory. Allocated: 0, Used by process: 14097043456 > [0] Memory requested 18446744069467867136 I'm using code from Firedrake Project (finite-element method) to discretize the equations. My code is available here: https://github.com/rbpiccinini/codes/blob/master/eigendrake/eigendrake_spe10_v2.py The code was run with 64GB available as RAM memory. The requested memory seems to be abnormally high, though. I'm able to solve for the eigenvalues of a 2d smaller problem. However, if I try to solve the 3d problem, I get the out of memory error. Could anybody help me understand the issue? Thanks in advance. -- Rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jul 1 18:26:23 2019 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 1 Jul 2019 19:26:23 -0400 Subject: [petsc-users] Out of memory - diffusion equation eigenvalues In-Reply-To: References: Message-ID: 3D uses a lot more memory. This is a crazy number (18446744069467867136). How large is your system? On Mon, Jul 1, 2019 at 7:20 PM Rodrigo Piccinini via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I'm trying to solve for the first ten smallest eigenvalues of a diffusion > equation in a 3d heterogeneous medium (varying diffusion coefficient), but > I'm getting an error message: > > Traceback (most recent call last): >> File "eigendrake_spe10_v2.py", line 121, in >> es.solve() >> File "SLEPc/EPS.pyx", line 1132, in slepc4py.SLEPc.EPS.solve >> petsc4py.PETSc.Error: error code 55 >> [0] EPSSolve() line 136 in >> /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssolve.c >> [0] EPSSetUp() line 269 in >> /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssetup.c >> [0] STSetUp() line 271 in >> /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/interface/stsolve.c >> [0] STSetUp_Shift() line 119 in >> /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/impls/shift/shift.c >> [0] KSPSetUp() line 391 in >> /tmp/pip-req-build-070lpqqd/src/ksp/ksp/interface/itfunc.c >> [0] PCSetUp() line 932 in >> /tmp/pip-req-build-070lpqqd/src/ksp/pc/interface/precon.c >> [0] PCSetUp_LU() line 99 in >> /tmp/pip-req-build-070lpqqd/src/ksp/pc/impls/factor/lu/lu.c >> [0] MatLUFactorSymbolic() line 3111 in >> /tmp/pip-req-build-070lpqqd/src/mat/interface/matrix.c >> [0] MatLUFactorSymbolic_SeqAIJ() line 366 in >> /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c >> [0] PetscMallocA() line 390 in >> /tmp/pip-req-build-070lpqqd/src/sys/memory/mal.c >> [0] MatLUFactorSymbolic_SeqAIJ() line 366 in >> /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c >> [0] Out of memory. Allocated: 0, Used by process: 14097043456 >> [0] Memory requested 18446744069467867136 > > > I'm using code from Firedrake Project > (finite-element method) to discretize the equations. > My code is available here: > https://github.com/rbpiccinini/codes/blob/master/eigendrake/eigendrake_spe10_v2.py > > The code was run with 64GB available as RAM memory. The requested memory > seems to be abnormally high, though. > I'm able to solve for the eigenvalues of a 2d smaller problem. However, if > I try to solve the 3d problem, I get the out of memory error. > > Could anybody help me understand the issue? > > Thanks in advance. > > -- > Rodrigo > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbpiccinini at gmail.com Mon Jul 1 19:07:57 2019 From: rbpiccinini at gmail.com (Rodrigo Piccinini) Date: Mon, 1 Jul 2019 21:07:57 -0300 Subject: [petsc-users] Out of memory - diffusion equation eigenvalues In-Reply-To: References: Message-ID: Hello, The number of unknowns after discretization is 60x85x220 = 1.12 million nodes. This should be the size of an eigenfunction vector in 3d. Em seg, 1 de jul de 2019 ?s 20:26, Mark Adams escreveu: > 3D uses a lot more memory. This is a crazy number (18446744069467867136). > How large is your system? > > On Mon, Jul 1, 2019 at 7:20 PM Rodrigo Piccinini via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello, >> >> I'm trying to solve for the first ten smallest eigenvalues of a diffusion >> equation in a 3d heterogeneous medium (varying diffusion coefficient), but >> I'm getting an error message: >> >> Traceback (most recent call last): >>> File "eigendrake_spe10_v2.py", line 121, in >>> es.solve() >>> File "SLEPc/EPS.pyx", line 1132, in slepc4py.SLEPc.EPS.solve >>> petsc4py.PETSc.Error: error code 55 >>> [0] EPSSolve() line 136 in >>> /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssolve.c >>> [0] EPSSetUp() line 269 in >>> /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssetup.c >>> [0] STSetUp() line 271 in >>> /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/interface/stsolve.c >>> [0] STSetUp_Shift() line 119 in >>> /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/impls/shift/shift.c >>> [0] KSPSetUp() line 391 in >>> /tmp/pip-req-build-070lpqqd/src/ksp/ksp/interface/itfunc.c >>> [0] PCSetUp() line 932 in >>> /tmp/pip-req-build-070lpqqd/src/ksp/pc/interface/precon.c >>> [0] PCSetUp_LU() line 99 in >>> /tmp/pip-req-build-070lpqqd/src/ksp/pc/impls/factor/lu/lu.c >>> [0] MatLUFactorSymbolic() line 3111 in >>> /tmp/pip-req-build-070lpqqd/src/mat/interface/matrix.c >>> [0] MatLUFactorSymbolic_SeqAIJ() line 366 in >>> /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c >>> [0] PetscMallocA() line 390 in >>> /tmp/pip-req-build-070lpqqd/src/sys/memory/mal.c >>> [0] MatLUFactorSymbolic_SeqAIJ() line 366 in >>> /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c >>> [0] Out of memory. Allocated: 0, Used by process: 14097043456 >>> [0] Memory requested 18446744069467867136 >> >> >> I'm using code from Firedrake Project >> (finite-element method) to discretize the equations. >> My code is available here: >> https://github.com/rbpiccinini/codes/blob/master/eigendrake/eigendrake_spe10_v2.py >> >> The code was run with 64GB available as RAM memory. The requested memory >> seems to be abnormally high, though. >> I'm able to solve for the eigenvalues of a 2d smaller problem. However, >> if I try to solve the 3d problem, I get the out of memory error. >> >> Could anybody help me understand the issue? >> >> Thanks in advance. >> >> -- >> Rodrigo >> > -- -- Rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: From hong at aspiritech.org Mon Jul 1 20:56:22 2019 From: hong at aspiritech.org (hong at aspiritech.org) Date: Mon, 1 Jul 2019 20:56:22 -0500 Subject: [petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE In-Reply-To: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> References: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> Message-ID: Ian: PETSc implementation of C = A*B requires C has same row ownership as A. I believe the distribution will be dominated by the dense matrices B and C, not sparse matrices A. Have you implemented C = A*B and logged performance? Hong Hi, > > I am recently trying to do matrix multiplication for C = A*B, where A is a > sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE. > > In matrix A, the nonzeros are not distributed evenly across the processor, > meaning that if using the default setting to let each processor own similar > number of rows, the number of nonzeros owned by each processor will be > significantly different. So I want to use different number of local rows > for each processor. In this case, does the MPIDense matrices B and C need > to be in the same row-layout as A? > > I mean, is something like the following is doable (A owns 3 rows and B, C > own 2 rows) > > > A B C > P0 o o o o | o. o o. o o > o o o o | o o o o o > o o o o | o. *. ----- = ---- > --------------- o o o o > P1 o o o o | o o o o o > > In this case, the entries can be evenly distributed for B and C thus more > memory efficient. > > But I am not sure would this make communication more complicated thus slow > down the overall wall time. How would you recommend to do? > a) Let rows of A and B be both evenly distributed > b) Let A have different rows layout, and B, C evenly distributed > c) Let A have different rows layout, and B, C follow A > > Or maybe other better way that I did not think about. > > Thanks a lot for your help, > Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jul 1 22:32:12 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 2 Jul 2019 03:32:12 +0000 Subject: [petsc-users] Out of memory - diffusion equation eigenvalues In-Reply-To: References: Message-ID: <7194B79F-1F69-454E-BC23-AC7D3FC377C8@anl.gov> This 18446744069467867136 suggests that you are using 32 bit indices but the factored matrix requires more than 2^31 nonzeros which cannot be represented with 32 but indices. You can try building PETSc with --with-64-bit-indices. You might also consider using SuperLU_DIST or MUMPS as your direct solver. They may be faster. Note SuperLUDIST works with 64 bit indices but MUMPS does not. Barry > On Jul 1, 2019, at 6:19 PM, Rodrigo Piccinini via petsc-users wrote: > > Hello, > > I'm trying to solve for the first ten smallest eigenvalues of a diffusion equation in a 3d heterogeneous medium (varying diffusion coefficient), but I'm getting an error message: > > Traceback (most recent call last): > File "eigendrake_spe10_v2.py", line 121, in > es.solve() > File "SLEPc/EPS.pyx", line 1132, in slepc4py.SLEPc.EPS.solve > petsc4py.PETSc.Error: error code 55 > [0] EPSSolve() line 136 in /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssolve.c > [0] EPSSetUp() line 269 in /tmp/pip-req-build-6t_4bkg3/src/eps/interface/epssetup.c > [0] STSetUp() line 271 in /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/interface/stsolve.c > [0] STSetUp_Shift() line 119 in /tmp/pip-req-build-6t_4bkg3/src/sys/classes/st/impls/shift/shift.c > [0] KSPSetUp() line 391 in /tmp/pip-req-build-070lpqqd/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 932 in /tmp/pip-req-build-070lpqqd/src/ksp/pc/interface/precon.c > [0] PCSetUp_LU() line 99 in /tmp/pip-req-build-070lpqqd/src/ksp/pc/impls/factor/lu/lu.c > [0] MatLUFactorSymbolic() line 3111 in /tmp/pip-req-build-070lpqqd/src/mat/interface/matrix.c > [0] MatLUFactorSymbolic_SeqAIJ() line 366 in /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c > [0] PetscMallocA() line 390 in /tmp/pip-req-build-070lpqqd/src/sys/memory/mal.c > [0] MatLUFactorSymbolic_SeqAIJ() line 366 in /tmp/pip-req-build-070lpqqd/src/mat/impls/aij/seq/aijfact.c > [0] Out of memory. Allocated: 0, Used by process: 14097043456 > [0] Memory requested 18446744069467867136 > > I'm using code from Firedrake Project (finite-element method) to discretize the equations. > My code is available here: https://github.com/rbpiccinini/codes/blob/master/eigendrake/eigendrake_spe10_v2.py > > The code was run with 64GB available as RAM memory. The requested memory seems to be abnormally high, though. > I'm able to solve for the eigenvalues of a 2d smaller problem. However, if I try to solve the 3d problem, I get the out of memory error. > > Could anybody help me understand the issue? > > Thanks in advance. > > -- > Rodrigo From amfoggia at gmail.com Tue Jul 2 03:35:28 2019 From: amfoggia at gmail.com (Ale Foggia) Date: Tue, 2 Jul 2019 10:35:28 +0200 Subject: [petsc-users] Communication during MatAssemblyEnd In-Reply-To: References: <845985ccc231460caea935e3e360c87c@DM6PR09MB3369.namprd09.prod.outlook.com> Message-ID: Thanks Jose for the help, also for me works now, and thank you Junchao for doing the test. I'll also think more from where this imbalance may come from, because there shouldn't be any, as far as I know. El lun., 1 jul. 2019 a las 18:36, Zhang, Junchao () escribi?: > Jose & Ale, > -ds_method 2 fixed the problem. I used PETSc master (f1480a5c) and > slepc master(675b89d7) through --download-slepc. I used > MKL /opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/ > I got the following results with 2048 processors. MatAssemblyEnd looks > expensive to me. I am looking into it. > > --- Event Stage 5: Offdiag > > BuildTwoSidedF 1 1.0 1.5201e+007345.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 13 0 0 0 0 0 > MatAssemblyBegin 1 1.0 1.5201e+005371.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 13 0 0 0 0 0 > MatAssemblyEnd 1 1.0 1.7720e+00 1.0 0.00e+00 0.0 5.7e+04 1.3e+05 > 8.0e+00 2 0 0 0 1 39 0100100100 0 > VecSet 1 1.0 2.4695e-02 7.9 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > --Junchao Zhang > > > On Mon, Jul 1, 2019 at 4:28 AM Jose E. Roman wrote: > >> You can try the following: >> - Try with a different DS method: -ds_method 1 or -ds_method 2 (see >> DSSetMethod) >> - Run with -ds_parallel synchronized (see DSSetParallel) >> If it does not help, send a reproducible code to slepc-maint >> >> Jose >> >> >> > El 1 jul 2019, a las 11:10, Ale Foggia via petsc-users < >> petsc-users at mcs.anl.gov> escribi?: >> > >> > Oh, I also got the same error when I switched to the newest version of >> SLEPc (using OpenBlas), and I don't know where it is coming from. >> > Can you tell me which version of SLEPc and PETSc are you using? And, >> are you using MKL? >> > Thanks for trying :) >> > >> > El vie., 28 jun. 2019 a las 16:57, Zhang, Junchao () >> escribi?: >> > Ran with 64 nodes and 32 ranks/node, met slepc errors and did not know >> how to proceed :( >> > >> > [363]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [363]PETSC ERROR: Error in external library >> > [363]PETSC ERROR: Error in LAPACK subroutine steqr: info=0 >> > [363]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> > [363]PETSC ERROR: Petsc Development GIT revision: >> v3.11.2-1052-gf1480a5c GIT Date: 2019-06-22 21:39:54 +0000 >> > [363]PETSC ERROR: /tmp/main.x on a arch-cray-xc40-knl-opt named >> nid03387 by jczhang Fri Jun 28 07:26:59 2019 >> > [1225]PETSC ERROR: #2 DSSolve() line 586 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c >> > [1225]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c >> > [1225]PETSC ERROR: #4 EPSSolve() line 149 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c >> > [240]PETSC ERROR: #2 DSSolve() line 586 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/sys/classes/ds/interface/dsops.c >> > [240]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 55 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/impls/krylov/krylovschur/ks-symm.c >> > [240]PETSC ERROR: #4 EPSSolve() line 149 in >> /global/u1/j/jczhang/petsc/arch-cray-xc40-knl-opt/externalpackages/git.slepc/src/eps/interface/epssolve.c >> > >> > --Junchao Zhang >> > >> > >> > On Fri, Jun 28, 2019 at 4:02 AM Ale Foggia wrote: >> > Junchao, >> > I'm sorry for the late response. >> > >> > El mi?., 26 jun. 2019 a las 16:39, Zhang, Junchao () >> escribi?: >> > Ale, >> > The job got a chance to run but failed with out-of-memory, "Some of >> your processes may have been killed by the cgroup out-of-memory handler." >> > >> > I mentioned that I used 1024 nodes and 32 processes on each node >> because the application needs a lot of memory. I think that for a system of >> size 38, one needs above 256 nodes for sure (assuming only 32 procs per >> node). I would try with 512 if it's possible. >> > >> > I also tried with 128 core with ./main.x 2 ... and got a weird error >> message "The size of the basis has to be at least equal to the number >> of MPI processes used." >> > >> > The error comes from the fact that you put a system size of only 2 >> which is too small. >> > I can also see the problem in the assembly with system sizes smaller >> than 38, so you can try with like 30 (for which I also have a log). In that >> case I run with 64 nodes and 32 processes per node. I think the problem may >> also fit in 32 nodes. >> > >> > --Junchao Zhang >> > >> > >> > On Tue, Jun 25, 2019 at 11:24 PM Junchao Zhang >> wrote: >> > Ale, >> > I successfully built your code and submitted a job to the NERSC Cori >> machine requiring 32768 KNL cores and one and a half hours. It is estimated >> to run in 3 days. If you also observed the same problem with less cores, >> what is your input arguments? Currently, I use what in your log file, >> ./main.x 38 -nn -j1 1.0 -d1 1.0 -eps_type krylovschur -eps_tol 1e-9 >> -log_view >> > The smaller the better. Thanks. >> > --Junchao Zhang >> > >> > >> > On Mon, Jun 24, 2019 at 6:20 AM Ale Foggia wrote: >> > Yes, I used KNL nodes. I you can perform the test would be great. Could >> it be that I'm not using the correct configuration of the KNL nodes? These >> are the environment variables I set: >> > MKL_NUM_THREADS=1 >> > OMP_NUM_THREADS=1 >> > KMP_HW_SUBSET=1t >> > KMP_AFFINITY=compact >> > I_MPI_PIN_DOMAIN=socket >> > I_MPI_PIN_PROCESSOR_LIST=0-63 >> > MKL_DYNAMIC=0 >> > >> > The code is in https://github.com/amfoggia/LSQuantumED and it has a >> readme to compile it and run it. When I run the test I used only 32 >> processors per node, and I used 1024 nodes in total, and it's for nspins=38. >> > Thank you >> > >> > El vie., 21 jun. 2019 a las 20:03, Zhang, Junchao () >> escribi?: >> > Ale, >> > Did you use Intel KNL nodes? Mr. Hong (cc'ed) did experiments on KNL >> nodes one year ago. He used 32768 processors and called MatAssemblyEnd 118 >> times and it used only 1.5 seconds in total. So I guess something was >> wrong with your test. If you can share your code, I can have a test on our >> machine to see how it goes. >> > Thanks. >> > --Junchao Zhang >> > >> > >> > On Fri, Jun 21, 2019 at 11:00 AM Junchao Zhang >> wrote: >> > MatAssembly was called once (in stage 5) and cost 2.5% of the total >> time. Look at stage 5. It says MatAssemblyBegin calls BuildTwoSidedF, >> which does global synchronization. The high max/min ratio means load >> imbalance. What I do not understand is MatAssemblyEnd. The ratio is 1.0. It >> means processors are already synchronized. With 32768 processors, there are >> 1.2e+06 messages with average length 1.9e+06 bytes. So each processor sends >> 36 (1.2e+06/32768) ~2MB messages and it takes 54 seconds. Another chance is >> the reduction at MatAssemblyEnd. I don't know why it needs 8 reductions. >> In my mind, one is enough. I need to look at the code. >> > >> > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages >> --- -- Message Lengths -- -- Reductions -- >> > Avg %Total Avg %Total Count >> %Total Avg %Total Count %Total >> > 0: Main Stage: 8.5045e+02 13.0% 3.0633e+15 14.0% 8.196e+07 >> 13.1% 7.768e+06 13.1% 2.530e+02 13.0% >> > 1: Create Basis: 7.9234e-02 0.0% 0.0000e+00 0.0% 0.000e+00 >> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >> > 2: Create Lattice: 8.3944e-05 0.0% 0.0000e+00 0.0% 0.000e+00 >> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >> > 3: Create Hamilt: 1.0694e+02 1.6% 0.0000e+00 0.0% 0.000e+00 >> 0.0% 0.000e+00 0.0% 2.000e+00 0.1% >> > 5: Offdiag: 1.6525e+02 2.5% 0.0000e+00 0.0% 1.188e+06 >> 0.2% 1.942e+06 0.0% 8.000e+00 0.4% >> > 6: Phys quantities: 5.4045e+03 82.8% 1.8866e+16 86.0% 5.417e+08 >> 86.7% 7.768e+06 86.8% 1.674e+03 86.1% >> > >> > --- Event Stage 5: Offdiag >> > BuildTwoSidedF 1 1.0 7.1565e+01 148448.9 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >> > MatAssemblyBegin 1 1.0 7.1565e+01 127783.7 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >> > MatAssemblyEnd 1 1.0 5.3762e+01 1.0 0.00e+00 0.0 1.2e+06 >> 1.9e+06 8.0e+00 1 0 0 0 0 33 0100100100 0 >> > VecSet 1 1.0 7.5533e-02 9.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > >> > >> > --Junchao Zhang >> > >> > >> > On Fri, Jun 21, 2019 at 10:34 AM Smith, Barry F. >> wrote: >> > >> > The load balance is definitely out of whack. >> > >> > >> > >> > BuildTwoSidedF 1 1.0 1.6722e-0241.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatMult 138 1.0 2.6604e+02 7.4 3.19e+10 2.1 8.2e+07 >> 7.8e+06 0.0e+00 2 4 13 13 0 15 25100100 0 2935476 >> > MatAssemblyBegin 1 1.0 1.6807e-0236.1 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatAssemblyEnd 1 1.0 3.5680e-01 3.9 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > VecNorm 2 1.0 4.4252e+0174.8 1.73e+07 1.0 0.0e+00 >> 0.0e+00 2.0e+00 1 0 0 0 0 5 0 0 0 1 12780 >> > VecCopy 6 1.0 6.5655e-02 2.6 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > VecAXPY 2 1.0 1.3793e-02 2.7 1.73e+07 1.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41000838 >> > VecScatterBegin 138 1.0 1.1653e+0285.8 0.00e+00 0.0 8.2e+07 >> 7.8e+06 0.0e+00 1 0 13 13 0 4 0100100 0 0 >> > VecScatterEnd 138 1.0 1.3653e+0222.4 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 >> > VecSetRandom 1 1.0 9.6668e-01 2.2 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > >> > Note that VecCopy/AXPY/SetRandom which are all embarrassingly parallel >> have a balance ratio above 2 which means some processes have more than >> twice the work of others. Meanwhile the ratio for anything with >> communication is extremely in balanced, some processes get to the >> synchronization point well before other processes. >> > >> > The first thing I would do is worry about the load imbalance, what is >> its cause? is it one process with much less work than others (not great but >> not terrible) or is it one process with much more work then the others >> (terrible) or something in between. I think once you get a handle on the >> load balance the rest may fall into place, otherwise we still have some >> exploring to do. This is not expected behavior for a good machine with a >> good network and a well balanced job. After you understand the load >> balancing you may need to use one of the parallel performance visualization >> tools to see why the synchronization is out of whack. >> > >> > Good luck >> > >> > Barry >> > >> > >> > > On Jun 21, 2019, at 9:27 AM, Ale Foggia wrote: >> > > >> > > I'm sending one with a bit less time. >> > > I'm timing the functions also with std::chronos and for the case of >> 180 seconds the program runs out of memory (and crushes) before the PETSc >> log gets to be printed, so I know the time only from my function. Anyway, >> in every case, the times between std::chronos and the PETSc log match. >> > > >> > > (The large times are in part "4b- Building offdiagonal part" or >> "Event Stage 5: Offdiag"). >> > > >> > > El vie., 21 jun. 2019 a las 16:09, Zhang, Junchao (< >> jczhang at mcs.anl.gov>) escribi?: >> > > >> > > >> > > On Fri, Jun 21, 2019 at 8:07 AM Ale Foggia >> wrote: >> > > Thanks both of you for your answers, >> > > >> > > El jue., 20 jun. 2019 a las 22:20, Smith, Barry F. (< >> bsmith at mcs.anl.gov>) escribi?: >> > > >> > > Note that this is a one time cost if the nonzero structure of the >> matrix stays the same. It will not happen in future MatAssemblies. >> > > >> > > > On Jun 20, 2019, at 3:16 PM, Zhang, Junchao via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > > > >> > > > Those messages were used to build MatMult communication pattern for >> the matrix. They were not part of the matrix entries-passing you imagined, >> but indeed happened in MatAssemblyEnd. If you want to make sure processors >> do not set remote entries, you can use >> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE), which will generate an >> error when an off-proc entry is set. >> > > >> > > I started being concerned about this when I saw that the assembly was >> taking a few hundreds of seconds in my code, like 180 seconds, which for me >> it's a considerable time. Do you think (or maybe you need more information >> to answer this) that this time is "reasonable" for communicating the >> pattern for the matrix? I already checked that I'm not setting any remote >> entries. >> > > It is not reasonable. Could you send log view of that test with 180 >> seconds MatAssembly? >> > > >> > > Also I see (in my code) that even if there are no messages being >> passed during MatAssemblyBegin, it is taking time and the "ratio" is very >> big. >> > > >> > > > >> > > > >> > > > --Junchao Zhang >> > > > >> > > > >> > > > On Thu, Jun 20, 2019 at 4:13 AM Ale Foggia via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > > > Hello all! >> > > > >> > > > During the conference I showed you a problem happening during >> MatAssemblyEnd in a particular code that I have. Now, I tried the same with >> a simple code (a symmetric problem corresponding to the Laplacian operator >> in 1D, from the SLEPc Hands-On exercises). As I understand (and please, >> correct me if I'm wrong), in this case the elements of the matrix are >> computed locally by each process so there should not be any communication >> during the assembly. However, in the log I get that there are messages >> being passed. Also, the number of messages changes with the number of >> processes used and the size of the matrix. Could you please help me >> understand this? >> > > > >> > > > I attach the code I used and the log I get for a small problem. >> > > > >> > > > Cheers, >> > > > Ale >> > > > >> > > >> > > >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From srahul_05 at yahoo.co.in Tue Jul 2 06:30:19 2019 From: srahul_05 at yahoo.co.in (Rahul Samala) Date: Tue, 2 Jul 2019 11:30:19 +0000 (UTC) Subject: [petsc-users] Solving a coupled multi-physics problem. References: <844125601.1861962.1562067019602.ref@mail.yahoo.com> Message-ID: <844125601.1861962.1562067019602@mail.yahoo.com> Hello PetSc users, I request suggestions on how to do a coupled problem, in my case coupling between multiphase flow in porous media and deformation of porous media.I have successfully written and validated the flow code with 3 degrees of freedom (S-saturation, P-pressure, T-Temperature) which uses SNES solver inside a time loop.? I have to use KSP solve for the deformation problem which has 1 degree of freedom (u-displacement). For the coupled problem, the flow part now requires the displacement vector and the deformation part requires the pressure field. I have written a pseudo-code below. My idea is to use the second solution vector (displacement) in FormFunctionLocal belonging to SNES. Since the DM used by SNES can be used for only one problem, a clone is made to be used for KSP. Is this approach the correct way to solve this coupled problem. typedef struct {?? ? ?? ?PetscScalar s; // Saturation ?? ?PetscScalar p; // Pressure ?? ?PetscScalar T; // Temperature } Field; typedef struct {?? ? ?? ?PetscScalar u; // Displacement } Field2; FormFunctionLocal(DMDALocalInfo *info, Field *sol, Field2 *sol2, Field *f, AppCtx *user) { ........................ } main() { ?? ?DM?? da, newda; ?? ?Vec? sol; ?? ?Vec? sol2; ?? ?SNES? snes; ?? ?KSP?? ksp; ?? ?SNESCreate(......, &snes); ?? ?DMDACreate1d(........,&da); ?? ?. ?? ?. ?? ?SNESSetDM(snes, da); ?? ?DMClone(da, &newda); ?? ?KSPCreate(.........,&ksp); ?? ?KSPSetDM(ksp, newda); ?? ?. ?? ?. ?? ?DMDASetUniformCoordinates(da,...........); ?? ?DMDASNESSetFunctionLocal(da,...........,FormFunctionLocal); ?? ?. ?? ?DMCreateGlobalVector(da, &sol); ?? ?DMCreateGlobalVector(newda, &sol2);?? ?// This sol2 of type Field2 is passed to FormFunctionLocal. ?? ?. ?? ?do { // Time loop ?? ? ?? ?SNESSolve(snes,....); ?? ?. ?? ?. ?? ?KSPSolve(ksp,......);?? ? ?? ?. ?? ?. ?? ?}while(criteria);? } Thank you,Rahul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From iancclin at umich.edu Tue Jul 2 07:37:15 2019 From: iancclin at umich.edu (Ian C. Lin) Date: Tue, 2 Jul 2019 08:37:15 -0400 Subject: [petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE In-Reply-To: References: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> Message-ID: <45BD8CEA-9108-45EA-9AD1-A9EF80444426@umich.edu> Dear Hong, Thanks for you suggestion. I have not implemented it yet as the re-distribution might involve some code changes to other part of the code, and I am not sure if that worth it. If the computation cost is mainly dominated by the distribution of the dense matrix and the efficiency won?t gain much, we might just avoid introducing this change. Currently, I printed out the nnzs owned by each processor, and the most one owns 60000 nnzs, and the least own owns 10000 nnzs, where the dimension of the matrix is 350000*350000. Do you have suggestions on the best approach? Thanks, Ian > hong at aspiritech.org ? 2019?7?1? 21:56 ??? > > Ian: > PETSc implementation of C = A*B requires C has same row ownership as A. > I believe the distribution will be dominated by the dense matrices B and C, not sparse matrices A. Have you implemented C = A*B and logged performance? > Hong > > Hi, > > I am recently trying to do matrix multiplication for C = A*B, where A is a sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE. > > In matrix A, the nonzeros are not distributed evenly across the processor, meaning that if using the default setting to let each processor own similar number of rows, the number of nonzeros owned by each processor will be significantly different. So I want to use different number of local rows for each processor. In this case, does the MPIDense matrices B and C need to be in the same row-layout as A? > > I mean, is something like the following is doable (A owns 3 rows and B, C own 2 rows) > > > A B C > P0 o o o o | o. o o. o o > o o o o | o o o o o > o o o o | o. *. ----- = ---- > --------------- o o o o > P1 o o o o | o o o o o > > In this case, the entries can be evenly distributed for B and C thus more memory efficient. > > But I am not sure would this make communication more complicated thus slow down the overall wall time. How would you recommend to do? > a) Let rows of A and B be both evenly distributed > b) Let A have different rows layout, and B, C evenly distributed > c) Let A have different rows layout, and B, C follow A > > Or maybe other better way that I did not think about. > > Thanks a lot for your help, > Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_mckinnell at aol.co.uk Tue Jul 2 03:24:42 2019 From: d_mckinnell at aol.co.uk (Daniel Mckinnell) Date: Tue, 2 Jul 2019 08:24:42 +0000 (UTC) Subject: [petsc-users] Altering Vertex Coordinates of an Unstructured Mesh References: <19378971.1107391.1562055882659.ref@mail.yahoo.com> Message-ID: <19378971.1107391.1562055882659@mail.yahoo.com> Hello,I am a student currently doing a summer project so am fairly new to Petsc. I am trying to alter vertex coordinates of an unstructured mesh to look at mesh movement in DMPlex after they have been created by DMPlexCreateBoxMesh. I have been altering the first example on the DMPlex documentation here: https://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/plex/examples/tutorials/ex1.c.html I have written something that vaguely does what I want in parallel, but it has a problem as the output from DMPlexGetPointLocal seems to be very inconsistent. Below is my code, I was wondering if anyone had a better way of doing this? I believeI have a bug but I am not sure where to start. Any help and advice/comments very welcome. Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: for_petsc_forum.cpp Type: text/x-c++src Size: 3853 bytes Desc: not available URL: From hong at aspiritech.org Tue Jul 2 09:09:21 2019 From: hong at aspiritech.org (hong at aspiritech.org) Date: Tue, 2 Jul 2019 09:09:21 -0500 Subject: [petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE In-Reply-To: <45BD8CEA-9108-45EA-9AD1-A9EF80444426@umich.edu> References: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> <45BD8CEA-9108-45EA-9AD1-A9EF80444426@umich.edu> Message-ID: Ian: > Thanks for you suggestion. I have not implemented it yet as the > re-distribution might involve some code changes to other part of the code, > and I am not sure if that worth it. If the computation cost is mainly > dominated by the distribution of the dense matrix and the efficiency won?t > gain much, we might just avoid introducing this change. Currently, I > printed out the nnzs owned by each processor, and the most one owns 60000 > nnzs, and the least own owns 10000 nnzs, where the dimension of the matrix > is 350000*350000. Do you have suggestions on the best approach? > The job balance also depends on many other factors, e.g., the connectivity of the matrix elements. I would start from the existing code, run it with '-log_view' and see the performance. Hong > hong at aspiritech.org ? 2019?7?1? 21:56 ??? > > Ian: > PETSc implementation of C = A*B requires C has same row ownership as A. > I believe the distribution will be dominated by the dense matrices B and > C, not sparse matrices A. Have you implemented C = A*B and logged > performance? > Hong > > Hi, >> >> I am recently trying to do matrix multiplication for C = A*B, where A is >> a sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE. >> >> In matrix A, the nonzeros are not distributed evenly across the >> processor, meaning that if using the default setting to let each processor >> own similar number of rows, the number of nonzeros owned by each processor >> will be significantly different. So I want to use different number of local >> rows for each processor. In this case, does the MPIDense matrices B and C >> need to be in the same row-layout as A? >> >> I mean, is something like the following is doable (A owns 3 rows and B, C >> own 2 rows) >> >> >> A B C >> P0 o o o o | o. o o. o o >> o o o o | o o o o o >> o o o o | o. *. ----- = ---- >> --------------- o o o o >> P1 o o o o | o o o o o >> >> In this case, the entries can be evenly distributed for B and C thus more >> memory efficient. >> >> But I am not sure would this make communication more complicated thus >> slow down the overall wall time. How would you recommend to do? >> a) Let rows of A and B be both evenly distributed >> b) Let A have different rows layout, and B, C evenly distributed >> c) Let A have different rows layout, and B, C follow A >> >> Or maybe other better way that I did not think about. >> >> Thanks a lot for your help, >> Ian > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 2 10:31:40 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 2 Jul 2019 15:31:40 +0000 Subject: [petsc-users] Solving a coupled multi-physics problem. In-Reply-To: <844125601.1861962.1562067019602@mail.yahoo.com> References: <844125601.1861962.1562067019602.ref@mail.yahoo.com> <844125601.1861962.1562067019602@mail.yahoo.com> Message-ID: <46A9658D-3A6D-429D-B5A5-3CCCF1B5C2AB@anl.gov> Rahul, This should be fine. But be careful since the original DMDA has run for 3 degrees of freedom per grid point while the u-displacement requires only one you will need to create a new DMDA identical to the first but with 1 dof (don't worry they and the parallel vectors they create will have the same parallel layout). What you propose is a splitting scheme which introduces additional errors due to the split off of the u-displacement. You could possibly also consider a fully implicit approach where the time-stepper/SNES see all 4 variables directly with no splitting. This would require additional (approximate) Jacobian terms that represent the coupling between the pressure/flow and u-displacement. If you do this you could use the higher implicit DAE/ODE integrators in PETSc instead of writing your own. Barry > On Jul 2, 2019, at 6:30 AM, Rahul Samala via petsc-users wrote: > > Hello PetSc users, > > I request suggestions on how to do a coupled problem, in my case coupling between multiphase flow in porous media and deformation of porous media. > I have successfully written and validated the flow code with 3 degrees of freedom (S-saturation, P-pressure, T-Temperature) > which uses SNES solver inside a time loop. I have to use KSP solve for the deformation problem which has 1 degree of freedom (u-displacement). > For the coupled problem, the flow part now requires the displacement vector and the deformation part requires > the pressure field. > > I have written a pseudo-code below. > My idea is to use the second solution vector (displacement) in FormFunctionLocal belonging to SNES. > Since the DM used by SNES can be used for only one problem, a clone is made to be used for KSP. > Is this approach the correct way to solve this coupled problem. > > typedef struct { > PetscScalar s; // Saturation > PetscScalar p; // Pressure > PetscScalar T; // Temperature > } Field; > > typedef struct { > PetscScalar u; // Displacement > } Field2; > > FormFunctionLocal(DMDALocalInfo *info, Field *sol, Field2 *sol2, Field *f, AppCtx *user) > { > ........................ > } > > main() > { > DM da, newda; > Vec sol; > Vec sol2; > SNES snes; > KSP ksp; > > SNESCreate(......, &snes); > DMDACreate1d(........,&da); > . > . > SNESSetDM(snes, da); > DMClone(da, &newda); > KSPCreate(.........,&ksp); > KSPSetDM(ksp, newda); > . > . > DMDASetUniformCoordinates(da,...........); > DMDASNESSetFunctionLocal(da,...........,FormFunctionLocal); > . > DMCreateGlobalVector(da, &sol); > DMCreateGlobalVector(newda, &sol2); // This sol2 of type Field2 is passed to FormFunctionLocal. > . > do { // Time loop > > SNESSolve(snes,....); > . > . > KSPSolve(ksp,......); > . > . > }while(criteria); > } > > Thank you, > Rahul. > > > > From bsmith at mcs.anl.gov Tue Jul 2 10:59:19 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 2 Jul 2019 15:59:19 +0000 Subject: [petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE In-Reply-To: <45BD8CEA-9108-45EA-9AD1-A9EF80444426@umich.edu> References: <9A12F922-0E3D-4947-ABF9-6A10442512F6@umich.edu> <45BD8CEA-9108-45EA-9AD1-A9EF80444426@umich.edu> Message-ID: Actually for sparse matrix - dense matrix product the work per process is proportional to the number of local nonzeros and the number of columns of the dense matrix so partitioning to get a close to equal number of nonzeros per process will even out the numerical work on each process. As Hong notes the time also depends on how many "ghost points" are needed to get the off-process parts of the dense matrix to the processes that need it. This depends on the sparsity pattern of the matrix. I would start by simply splitting the rows of the sparse matrix to get roughly the same number of nonzeros on each process. Barry > On Jul 2, 2019, at 7:37 AM, Ian C. Lin via petsc-users wrote: > > Dear Hong, > > Thanks for you suggestion. I have not implemented it yet as the re-distribution might involve some code changes to other part of the code, and I am not sure if that worth it. If the computation cost is mainly dominated by the distribution of the dense matrix and the efficiency won?t gain much, we might just avoid introducing this change. Currently, I printed out the nnzs owned by each processor, and the most one owns 60000 nnzs, and the least own owns 10000 nnzs, where the dimension of the matrix is 350000*350000. Do you have suggestions on the best approach? > > Thanks, > Ian > >> hong at aspiritech.org ? 2019?7?1? 21:56 ??? >> >> Ian: >> PETSc implementation of C = A*B requires C has same row ownership as A. >> I believe the distribution will be dominated by the dense matrices B and C, not sparse matrices A. Have you implemented C = A*B and logged performance? >> Hong >> >> Hi, >> >> I am recently trying to do matrix multiplication for C = A*B, where A is a sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE. >> >> In matrix A, the nonzeros are not distributed evenly across the processor, meaning that if using the default setting to let each processor own similar number of rows, the number of nonzeros owned by each processor will be significantly different. So I want to use different number of local rows for each processor. In this case, does the MPIDense matrices B and C need to be in the same row-layout as A? >> >> I mean, is something like the following is doable (A owns 3 rows and B, C own 2 rows) >> >> >> A B C >> P0 o o o o | o. o o. o o >> o o o o | o o o o o >> o o o o | o. *. ----- = ---- >> --------------- o o o o >> P1 o o o o | o o o o o >> >> In this case, the entries can be evenly distributed for B and C thus more memory efficient. >> >> But I am not sure would this make communication more complicated thus slow down the overall wall time. How would you recommend to do? >> a) Let rows of A and B be both evenly distributed >> b) Let A have different rows layout, and B, C evenly distributed >> c) Let A have different rows layout, and B, C follow A >> >> Or maybe other better way that I did not think about. >> >> Thanks a lot for your help, >> Ian > From epscodes at gmail.com Tue Jul 2 21:40:01 2019 From: epscodes at gmail.com (Xiangdong) Date: Tue, 2 Jul 2019 22:40:01 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof Message-ID: Hello everyone, When I run the ex19 with cuda like this: mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view it worked fine and produced correct results. However, when I tried to run this example with nvprof: nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view I got errors like: [3]PETSC ERROR: Error in external library [3]PETSC ERROR: CUDA error 700 [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 2019 [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda-dir=/home/eps/MyLocal/cuda --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install --download-hdf5=1 [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu [3]PETSC ERROR: #2 VecSet() line 547 in /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu [3]PETSC ERROR: #4 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu [3]PETSC ERROR: #6 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in /home/eps/MyLocal/petsc/src/mat/interface/matrix.c [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c [3]PETSC ERROR: #12 SNESSolve() line 4560 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c The full run log is attached. I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1. I do not know why it is okay without nvprof but crashed with nvprof. Any suggestion to fix this? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- eps at new:~/MyCodes/petsctest$ nvprof --profile-child-processes mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4979== NVPROF is profiling process 4979, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4980== NVPROF is profiling process 4980, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4981== NVPROF is profiling process 4981, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4978== NVPROF is profiling process 4978, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Error in external library [3]PETSC ERROR: CUDA error 700 [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 2019 [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda-dir=/home/eps/MyLocal/cuda --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install --download-hdf5=1 [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu [3]PETSC ERROR: #2 VecSet() line 547 in /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu [3]PETSC ERROR: #4 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu [3]PETSC ERROR: #6 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in /home/eps/MyLocal/petsc/src/mat/interface/matrix.c [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c [3]PETSC ERROR: #12 SNESSolve() line 4560 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c [3]PETSC ERROR: PETSc Option Table entries: [3]PETSC ERROR: -da_refine 5 [3]PETSC ERROR: -ksp_monitor [3]PETSC ERROR: -log_view [3]PETSC ERROR: -mat_type aijcusparse [3]PETSC ERROR: -snes_monitor [3]PETSC ERROR: -snes_view [3]PETSC ERROR: -vec_type cuda [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3 ==4981== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4981== Profiling result: No kernels were profiled. No API activities were profiled. ==4981== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. ==4979== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4979== Profiling result: No kernels were profiled. No API activities were profiled. ==4979== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. ==4980== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4980== Profiling result: No kernels were profiled. No API activities were profiled. ==4980== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. ==4978== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view ==4978== Profiling result: No kernels were profiled. No API activities were profiled. ==4978== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. ======== Error: Application returned non-zero code 76 From knepley at gmail.com Wed Jul 3 06:50:04 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Jul 2019 06:50:04 -0500 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: On Tue, Jul 2, 2019 at 9:41 PM Xiangdong via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello everyone, > > When I run the ex19 with cuda like this: > mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor > -mat_type aijcusparse -vec_type cuda -log_view > > it worked fine and produced correct results. > > However, when I tried to run this example with nvprof: > nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 > -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda > -log_view > > I got errors like: > [3]PETSC ERROR: Error in external library > [3]PETSC ERROR: CUDA error 700 > Looking at Google for this error, it seems that it tends to arise with incompatible drivers. Perhaps your nvprof is older/newer than the library PETSc is linked against? Matt > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown > [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 > 2019 > [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 > --with-mpi-dir=/home/eps/MyLocal/mpi/mpich > --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl > --with-cuda-dir=/home/eps/MyLocal/cuda > --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install > --download-hdf5=1 > [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [3]PETSC ERROR: #2 VecSet() line 547 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c > [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #4 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #6 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in > /home/eps/MyLocal/petsc/src/mat/interface/matrix.c > [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in > /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c > [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in > /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c > [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in > /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c > [3]PETSC ERROR: #12 SNESSolve() line 4560 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > > The full run log is attached. > > I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA > Version: 10.1. > > I do not know why it is okay without nvprof but crashed with nvprof. Any > suggestion to fix this? > > Thank you. > > Best, > Xiangdong > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Wed Jul 3 07:36:03 2019 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Wed, 3 Jul 2019 14:36:03 +0200 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran Message-ID: Dear Matt, Dear Petsc After my recent pull of petsc master I encounter test fails that use dmplex meshes. I attached a simple script to show an issue where DMPlexRestoreCone mangles the cone relationships in the DM. The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 We recently had a system update and I am not 100% sure if this is some weird stuff going on at my end or if this is indeed Plex related or if I should not call Restore in the inner loop. Anyway, I would be most grateful if you could try to reproduce the error and hint me towards a solution. As always, many many thanks! Fabian P.S. here the output of make run and note the cones, the second time they are outputted... ./plex_prism -show_plex ::ascii_info_detail DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 1 (0) [0]: 0 <---- 2 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 6 (0) [0]: 1 <---- 7 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 12 (0) [0]: 2 <---- 13 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 7 (0) [0]: 3 <---- 9 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 6 (0) [0]: 4 <---- 9 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 8 (0) [0]: 5 <---- 10 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 15 (0) [0]: 6 <---- 16 (0) [0]: 7 <---- 15 (0) [0]: 7 <---- 17 (0) [0]: 8 <---- 16 (0) [0]: 8 <---- 17 (0) [0]: 9 <---- 15 (0) [0]: 9 <---- 18 (0) [0]: 10 <---- 16 (0) [0]: 10 <---- 19 (0) [0]: 11 <---- 17 (0) [0]: 11 <---- 20 (0) [0]: 12 <---- 18 (0) [0]: 12 <---- 19 (0) [0]: 13 <---- 18 (0) [0]: 13 <---- 20 (0) [0]: 14 <---- 19 (0) [0]: 14 <---- 20 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 0 (0) [0]: 0 <---- 0 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 0 (0) [0]: 1 <---- 0 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 0 (0) [0]: 2 <---- 0 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 0 (0) [0]: 3 <---- 0 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 0 (0) [0]: 4 <---- 0 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 0 (0) [0]: 5 <---- 0 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 0 (0) [0]: 6 <---- 0 (0) [0]: 7 <---- 0 (0) [0]: 7 <---- 0 (0) [0]: 8 <---- 0 (0) [0]: 8 <---- 0 (0) [0]: 9 <---- 0 (0) [0]: 9 <---- 0 (0) [0]: 10 <---- 0 (0) [0]: 10 <---- 0 (0) [0]: 11 <---- 0 (0) [0]: 11 <---- 0 (0) [0]: 12 <---- 0 (0) [0]: 12 <---- 0 (0) [0]: 13 <---- 0 (0) [0]: 13 <---- 0 (0) [0]: 14 <---- 0 (0) [0]: 14 <---- 0 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. -------------- next part -------------- ./plex_prism -show_plex ::ascii_info_detail DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 1 (0) [0]: 0 <---- 2 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 6 (0) [0]: 1 <---- 7 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 12 (0) [0]: 2 <---- 13 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 7 (0) [0]: 3 <---- 9 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 6 (0) [0]: 4 <---- 9 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 8 (0) [0]: 5 <---- 10 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 15 (0) [0]: 6 <---- 16 (0) [0]: 7 <---- 15 (0) [0]: 7 <---- 17 (0) [0]: 8 <---- 16 (0) [0]: 8 <---- 17 (0) [0]: 9 <---- 15 (0) [0]: 9 <---- 18 (0) [0]: 10 <---- 16 (0) [0]: 10 <---- 19 (0) [0]: 11 <---- 17 (0) [0]: 11 <---- 20 (0) [0]: 12 <---- 18 (0) [0]: 12 <---- 19 (0) [0]: 13 <---- 18 (0) [0]: 13 <---- 20 (0) [0]: 14 <---- 19 (0) [0]: 14 <---- 20 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 0 (0) [0]: 0 <---- 0 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 0 (0) [0]: 1 <---- 0 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 0 (0) [0]: 2 <---- 0 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 0 (0) [0]: 3 <---- 0 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 0 (0) [0]: 4 <---- 0 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 0 (0) [0]: 5 <---- 0 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 0 (0) [0]: 6 <---- 0 (0) [0]: 7 <---- 0 (0) [0]: 7 <---- 0 (0) [0]: 8 <---- 0 (0) [0]: 8 <---- 0 (0) [0]: 9 <---- 0 (0) [0]: 9 <---- 0 (0) [0]: 10 <---- 0 (0) [0]: 10 <---- 0 (0) [0]: 11 <---- 0 (0) [0]: 11 <---- 0 (0) [0]: 12 <---- 0 (0) [0]: 12 <---- 0 (0) [0]: 13 <---- 0 (0) [0]: 13 <---- 0 (0) [0]: 14 <---- 0 (0) [0]: 14 <---- 0 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. Cone 0 : 1 2 3 4 5 Cone 1 : 6 7 8 Cone 2 : 12 13 14 Cone 3 : 7 9 11 13 Cone 4 : 6 9 10 12 Cone 5 : 8 10 11 14 Cone 6 : 15 16 Cone 7 : 15 17 Cone 8 : 16 17 Cone 9 : 15 18 Cone 10 : 16 19 Cone 11 : 17 20 Cone 12 : 18 19 Cone 13 : 18 20 Cone 14 : 19 20 Cone 15 : Cone 16 : Cone 17 : Cone 18 : Cone 19 : Cone 20 : -------------- next part -------------- include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules run:: plex_prism ./plex_prism -show_plex ::ascii_info_detail plex_prism:: plex_prism.F90 ${PETSC_FCOMPILE} -c plex_prism.F90 ${FLINKER} plex_prism.o -o plex_prism ${PETSC_LIB} clean:: rm -rf *.o prism.h5 prism.xmf plex_prism -------------- next part -------------- A non-text attachment was scrubbed... Name: plex_prism.F90 Type: text/x-fortran Size: 6058 bytes Desc: not available URL: From D.Liu-4 at tudelft.nl Wed Jul 3 08:10:50 2019 From: D.Liu-4 at tudelft.nl (Dongyu Liu - CITG) Date: Wed, 3 Jul 2019 13:10:50 +0000 Subject: [petsc-users] How to create a mapping of global to local indices? Message-ID: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> Hi, I am running a FEM program using PETSc. In the beginning, the mesh is partitioned in gmsh, and we read the partitioned mesh using our own reader. Now my question is: How can I get a global to local indices mapping? and do I need to renumber the indices after I read the partitioned mesh? >From my understanding, the rows handled by each processor should be consecutive, but the indices of nodes which I directly read from a gmsh file are not. Is this an issue? Best, Dongyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 3 08:18:05 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Jul 2019 08:18:05 -0500 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: References: Message-ID: On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear Matt, Dear Petsc > > After my recent pull of petsc master I encounter test fails that use > dmplex meshes. > > I attached a simple script to show an issue where DMPlexRestoreCone > mangles the cone relationships in the DM. > > The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 > @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 > > We recently had a system update and I am not 100% sure if this is some > weird stuff going on at my end or if this is indeed Plex related or if I > should not call Restore in the inner loop. > > Anyway, I would be most grateful if you could try to reproduce the error > and hint me towards a solution. > Hi Fabian, I think it might be your system update. Seems to run fine for me: master *:~/Downloads/tmp$ ./plex_prism Cone 0 : 1 2 3 4 5 Cone 1 : 6 7 8 Cone 2 : 12 13 14 Cone 3 : 7 9 11 13 Cone 4 : 6 9 10 12 Cone 5 : 8 10 11 14 Cone 6 : 15 16 Cone 7 : 15 17 Cone 8 : 16 17 Cone 9 : 15 18 Cone 10 : 16 19 Cone 11 : 17 20 Cone 12 : 18 19 Cone 13 : 18 20 Cone 14 : 19 20 Cone 15 : Cone 16 : Cone 17 : Cone 18 : Cone 19 : Cone 20 : master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 1 (0) [0]: 0 <---- 2 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 6 (0) [0]: 1 <---- 7 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 12 (0) [0]: 2 <---- 13 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 7 (0) [0]: 3 <---- 9 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 6 (0) [0]: 4 <---- 9 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 8 (0) [0]: 5 <---- 10 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 15 (0) [0]: 6 <---- 16 (0) [0]: 7 <---- 15 (0) [0]: 7 <---- 17 (0) [0]: 8 <---- 16 (0) [0]: 8 <---- 17 (0) [0]: 9 <---- 15 (0) [0]: 9 <---- 18 (0) [0]: 10 <---- 16 (0) [0]: 10 <---- 19 (0) [0]: 11 <---- 17 (0) [0]: 11 <---- 20 (0) [0]: 12 <---- 18 (0) [0]: 12 <---- 19 (0) [0]: 13 <---- 18 (0) [0]: 13 <---- 20 (0) [0]: 14 <---- 19 (0) [0]: 14 <---- 20 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. Cone 0 : 1 2 3 4 5 Cone 1 : 6 7 8 Cone 2 : 12 13 14 Cone 3 : 7 9 11 13 Cone 4 : 6 9 10 12 Cone 5 : 8 10 11 14 Cone 6 : 15 16 Cone 7 : 15 17 Cone 8 : 16 17 Cone 9 : 15 18 Cone 10 : 16 19 Cone 11 : 17 20 Cone 12 : 18 19 Cone 13 : 18 20 Cone 14 : 19 20 Cone 15 : Cone 16 : Cone 17 : Cone 18 : Cone 19 : Cone 20 : DM Object: testplex 1 MPI processes type: plex testplex in 3 dimensions: Supports: [0] Max support size: 3 [0]: 1 ----> 0 [0]: 2 ----> 0 [0]: 3 ----> 0 [0]: 4 ----> 0 [0]: 5 ----> 0 [0]: 6 ----> 1 [0]: 6 ----> 4 [0]: 7 ----> 1 [0]: 7 ----> 3 [0]: 8 ----> 1 [0]: 8 ----> 5 [0]: 9 ----> 3 [0]: 9 ----> 4 [0]: 10 ----> 4 [0]: 10 ----> 5 [0]: 11 ----> 3 [0]: 11 ----> 5 [0]: 12 ----> 2 [0]: 12 ----> 4 [0]: 13 ----> 2 [0]: 13 ----> 3 [0]: 14 ----> 2 [0]: 14 ----> 5 [0]: 15 ----> 6 [0]: 15 ----> 7 [0]: 15 ----> 9 [0]: 16 ----> 6 [0]: 16 ----> 8 [0]: 16 ----> 10 [0]: 17 ----> 7 [0]: 17 ----> 8 [0]: 17 ----> 11 [0]: 18 ----> 9 [0]: 18 ----> 12 [0]: 18 ----> 13 [0]: 19 ----> 10 [0]: 19 ----> 12 [0]: 19 ----> 14 [0]: 20 ----> 11 [0]: 20 ----> 13 [0]: 20 ----> 14 Cones: [0] Max cone size: 5 [0]: 0 <---- 1 (0) [0]: 0 <---- 2 (0) [0]: 0 <---- 3 (0) [0]: 0 <---- 4 (0) [0]: 0 <---- 5 (0) [0]: 1 <---- 6 (0) [0]: 1 <---- 7 (0) [0]: 1 <---- 8 (0) [0]: 2 <---- 12 (0) [0]: 2 <---- 13 (0) [0]: 2 <---- 14 (0) [0]: 3 <---- 7 (0) [0]: 3 <---- 9 (0) [0]: 3 <---- 11 (0) [0]: 3 <---- 13 (0) [0]: 4 <---- 6 (0) [0]: 4 <---- 9 (0) [0]: 4 <---- 10 (0) [0]: 4 <---- 12 (0) [0]: 5 <---- 8 (0) [0]: 5 <---- 10 (0) [0]: 5 <---- 11 (0) [0]: 5 <---- 14 (0) [0]: 6 <---- 15 (0) [0]: 6 <---- 16 (0) [0]: 7 <---- 15 (0) [0]: 7 <---- 17 (0) [0]: 8 <---- 16 (0) [0]: 8 <---- 17 (0) [0]: 9 <---- 15 (0) [0]: 9 <---- 18 (0) [0]: 10 <---- 16 (0) [0]: 10 <---- 19 (0) [0]: 11 <---- 17 (0) [0]: 11 <---- 20 (0) [0]: 12 <---- 18 (0) [0]: 12 <---- 19 (0) [0]: 13 <---- 18 (0) [0]: 13 <---- 20 (0) [0]: 14 <---- 19 (0) [0]: 14 <---- 20 (0) coordinates with 1 fields field 0 with 3 components Process 0: ( 0) dim 0 offset 0 ( 1) dim 0 offset 0 ( 2) dim 0 offset 0 ( 3) dim 0 offset 0 ( 4) dim 0 offset 0 ( 5) dim 0 offset 0 ( 6) dim 0 offset 0 ( 7) dim 0 offset 0 ( 8) dim 0 offset 0 ( 9) dim 0 offset 0 ( 10) dim 0 offset 0 ( 11) dim 0 offset 0 ( 12) dim 0 offset 0 ( 13) dim 0 offset 0 ( 14) dim 0 offset 0 ( 15) dim 3 offset 0 0. 0. 0. ( 16) dim 3 offset 3 0. 2. 0. ( 17) dim 3 offset 6 2. 1. 0. ( 18) dim 3 offset 9 0. 0. 1. ( 19) dim 3 offset 12 0. 2. 1. ( 20) dim 3 offset 15 2. 1. 1. Thanks, Matt > As always, many many thanks! > > Fabian > > P.S. here the output of > make run > > and note the cones, the second time they are outputted... > > ./plex_prism -show_plex ::ascii_info_detail > DM Object: testplex 1 MPI processes > type: plex > testplex in 3 dimensions: > Supports: > [0] Max support size: 3 > [0]: 1 ----> 0 > [0]: 2 ----> 0 > [0]: 3 ----> 0 > [0]: 4 ----> 0 > [0]: 5 ----> 0 > [0]: 6 ----> 1 > [0]: 6 ----> 4 > [0]: 7 ----> 1 > [0]: 7 ----> 3 > [0]: 8 ----> 1 > [0]: 8 ----> 5 > [0]: 9 ----> 3 > [0]: 9 ----> 4 > [0]: 10 ----> 4 > [0]: 10 ----> 5 > [0]: 11 ----> 3 > [0]: 11 ----> 5 > [0]: 12 ----> 2 > [0]: 12 ----> 4 > [0]: 13 ----> 2 > [0]: 13 ----> 3 > [0]: 14 ----> 2 > [0]: 14 ----> 5 > [0]: 15 ----> 6 > [0]: 15 ----> 7 > [0]: 15 ----> 9 > [0]: 16 ----> 6 > [0]: 16 ----> 8 > [0]: 16 ----> 10 > [0]: 17 ----> 7 > [0]: 17 ----> 8 > [0]: 17 ----> 11 > [0]: 18 ----> 9 > [0]: 18 ----> 12 > [0]: 18 ----> 13 > [0]: 19 ----> 10 > [0]: 19 ----> 12 > [0]: 19 ----> 14 > [0]: 20 ----> 11 > [0]: 20 ----> 13 > [0]: 20 ----> 14 > Cones: > [0] Max cone size: 5 > [0]: 0 <---- 1 (0) > [0]: 0 <---- 2 (0) > [0]: 0 <---- 3 (0) > [0]: 0 <---- 4 (0) > [0]: 0 <---- 5 (0) > [0]: 1 <---- 6 (0) > [0]: 1 <---- 7 (0) > [0]: 1 <---- 8 (0) > [0]: 2 <---- 12 (0) > [0]: 2 <---- 13 (0) > [0]: 2 <---- 14 (0) > [0]: 3 <---- 7 (0) > [0]: 3 <---- 9 (0) > [0]: 3 <---- 11 (0) > [0]: 3 <---- 13 (0) > [0]: 4 <---- 6 (0) > [0]: 4 <---- 9 (0) > [0]: 4 <---- 10 (0) > [0]: 4 <---- 12 (0) > [0]: 5 <---- 8 (0) > [0]: 5 <---- 10 (0) > [0]: 5 <---- 11 (0) > [0]: 5 <---- 14 (0) > [0]: 6 <---- 15 (0) > [0]: 6 <---- 16 (0) > [0]: 7 <---- 15 (0) > [0]: 7 <---- 17 (0) > [0]: 8 <---- 16 (0) > [0]: 8 <---- 17 (0) > [0]: 9 <---- 15 (0) > [0]: 9 <---- 18 (0) > [0]: 10 <---- 16 (0) > [0]: 10 <---- 19 (0) > [0]: 11 <---- 17 (0) > [0]: 11 <---- 20 (0) > [0]: 12 <---- 18 (0) > [0]: 12 <---- 19 (0) > [0]: 13 <---- 18 (0) > [0]: 13 <---- 20 (0) > [0]: 14 <---- 19 (0) > [0]: 14 <---- 20 (0) > coordinates with 1 fields > field 0 with 3 components > Process 0: > ( 0) dim 0 offset 0 > ( 1) dim 0 offset 0 > ( 2) dim 0 offset 0 > ( 3) dim 0 offset 0 > ( 4) dim 0 offset 0 > ( 5) dim 0 offset 0 > ( 6) dim 0 offset 0 > ( 7) dim 0 offset 0 > ( 8) dim 0 offset 0 > ( 9) dim 0 offset 0 > ( 10) dim 0 offset 0 > ( 11) dim 0 offset 0 > ( 12) dim 0 offset 0 > ( 13) dim 0 offset 0 > ( 14) dim 0 offset 0 > ( 15) dim 3 offset 0 0. 0. 0. > ( 16) dim 3 offset 3 0. 2. 0. > ( 17) dim 3 offset 6 2. 1. 0. > ( 18) dim 3 offset 9 0. 0. 1. > ( 19) dim 3 offset 12 0. 2. 1. > ( 20) dim 3 offset 15 2. 1. 1. > > > > > > > > > > DM Object: testplex 1 MPI processes > type: plex > testplex in 3 dimensions: > Supports: > [0] Max support size: 3 > [0]: 1 ----> 0 > [0]: 2 ----> 0 > [0]: 3 ----> 0 > [0]: 4 ----> 0 > [0]: 5 ----> 0 > [0]: 6 ----> 1 > [0]: 6 ----> 4 > [0]: 7 ----> 1 > [0]: 7 ----> 3 > [0]: 8 ----> 1 > [0]: 8 ----> 5 > [0]: 9 ----> 3 > [0]: 9 ----> 4 > [0]: 10 ----> 4 > [0]: 10 ----> 5 > [0]: 11 ----> 3 > [0]: 11 ----> 5 > [0]: 12 ----> 2 > [0]: 12 ----> 4 > [0]: 13 ----> 2 > [0]: 13 ----> 3 > [0]: 14 ----> 2 > [0]: 14 ----> 5 > [0]: 15 ----> 6 > [0]: 15 ----> 7 > [0]: 15 ----> 9 > [0]: 16 ----> 6 > [0]: 16 ----> 8 > [0]: 16 ----> 10 > [0]: 17 ----> 7 > [0]: 17 ----> 8 > [0]: 17 ----> 11 > [0]: 18 ----> 9 > [0]: 18 ----> 12 > [0]: 18 ----> 13 > [0]: 19 ----> 10 > [0]: 19 ----> 12 > [0]: 19 ----> 14 > [0]: 20 ----> 11 > [0]: 20 ----> 13 > [0]: 20 ----> 14 > Cones: > [0] Max cone size: 5 > [0]: 0 <---- 0 (0) > [0]: 0 <---- 0 (0) > [0]: 0 <---- 3 (0) > [0]: 0 <---- 4 (0) > [0]: 0 <---- 5 (0) > [0]: 1 <---- 0 (0) > [0]: 1 <---- 0 (0) > [0]: 1 <---- 8 (0) > [0]: 2 <---- 0 (0) > [0]: 2 <---- 0 (0) > [0]: 2 <---- 14 (0) > [0]: 3 <---- 0 (0) > [0]: 3 <---- 0 (0) > [0]: 3 <---- 11 (0) > [0]: 3 <---- 13 (0) > [0]: 4 <---- 0 (0) > [0]: 4 <---- 0 (0) > [0]: 4 <---- 10 (0) > [0]: 4 <---- 12 (0) > [0]: 5 <---- 0 (0) > [0]: 5 <---- 0 (0) > [0]: 5 <---- 11 (0) > [0]: 5 <---- 14 (0) > [0]: 6 <---- 0 (0) > [0]: 6 <---- 0 (0) > [0]: 7 <---- 0 (0) > [0]: 7 <---- 0 (0) > [0]: 8 <---- 0 (0) > [0]: 8 <---- 0 (0) > [0]: 9 <---- 0 (0) > [0]: 9 <---- 0 (0) > [0]: 10 <---- 0 (0) > [0]: 10 <---- 0 (0) > [0]: 11 <---- 0 (0) > [0]: 11 <---- 0 (0) > [0]: 12 <---- 0 (0) > [0]: 12 <---- 0 (0) > [0]: 13 <---- 0 (0) > [0]: 13 <---- 0 (0) > [0]: 14 <---- 0 (0) > [0]: 14 <---- 0 (0) > coordinates with 1 fields > field 0 with 3 components > Process 0: > ( 0) dim 0 offset 0 > ( 1) dim 0 offset 0 > ( 2) dim 0 offset 0 > ( 3) dim 0 offset 0 > ( 4) dim 0 offset 0 > ( 5) dim 0 offset 0 > ( 6) dim 0 offset 0 > ( 7) dim 0 offset 0 > ( 8) dim 0 offset 0 > ( 9) dim 0 offset 0 > ( 10) dim 0 offset 0 > ( 11) dim 0 offset 0 > ( 12) dim 0 offset 0 > ( 13) dim 0 offset 0 > ( 14) dim 0 offset 0 > ( 15) dim 3 offset 0 0. 0. 0. > ( 16) dim 3 offset 3 0. 2. 0. > ( 17) dim 3 offset 6 2. 1. 0. > ( 18) dim 3 offset 9 0. 0. 1. > ( 19) dim 3 offset 12 0. 2. 1. > ( 20) dim 3 offset 15 2. 1. 1. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 3 08:54:49 2019 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 3 Jul 2019 09:54:49 -0400 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> Message-ID: PETSc matrices and vectors are created with a local size n or global size N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from 0 to n_0 - 1 where n_0 is the number of equations on process 0. This numbering continues for all processes. You can use: PetscErrorCode MatGetOwnershipRange (Mat mat,PetscInt *m,PetscInt *n) To see what your local global indices are (rows m to n-1 are on this process and so its the local indices are 0 to (n-m-1)). Mark On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi, > > > I am running a FEM program using PETSc. In the beginning, the mesh is > partitioned in gmsh, and we read the partitioned mesh using our own reader. > > > Now my question is: How can I get a global to local indices mapping? and > do I need to renumber the indices after I read the partitioned mesh? > > > From my understanding, the rows handled by each processor should be > consecutive, but the indices of nodes which I directly read from a gmsh > file are not. Is this an issue? > > > Best, > > > Dongyu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 3 09:00:05 2019 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 3 Jul 2019 10:00:05 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: I get this error on normal runs on SUMMIT when I have an empty process. It does not happen all of the time but it does look to be deterministic. I'm debugging it now. On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello everyone, > > When I run the ex19 with cuda like this: > mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor > -mat_type aijcusparse -vec_type cuda -log_view > > it worked fine and produced correct results. > > However, when I tried to run this example with nvprof: > nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 > -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda > -log_view > > I got errors like: > [3]PETSC ERROR: Error in external library > [3]PETSC ERROR: CUDA error 700 > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown > [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 > 2019 > [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 > --with-mpi-dir=/home/eps/MyLocal/mpi/mpich > --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl > --with-cuda-dir=/home/eps/MyLocal/cuda > --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install > --download-hdf5=1 > [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [3]PETSC ERROR: #2 VecSet() line 547 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c > [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #4 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #6 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in > /home/eps/MyLocal/petsc/src/mat/interface/matrix.c > [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in > /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c > [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in > /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c > [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in > /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c > [3]PETSC ERROR: #12 SNESSolve() line 4560 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > > The full run log is attached. > > I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA > Version: 10.1. > > I do not know why it is okay without nvprof but crashed with nvprof. Any > suggestion to fix this? > > Thank you. > > Best, > Xiangdong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From D.Liu-4 at tudelft.nl Wed Jul 3 09:06:33 2019 From: D.Liu-4 at tudelft.nl (Dongyu Liu - CITG) Date: Wed, 3 Jul 2019 14:06:33 +0000 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl>, Message-ID: <963a6ee9947046a6909192b369cd44b4@tudelft.nl> Thank you, Mark. My question is if I have some node indices like [0, 1, 3, 8, 9] handled by the current processor, is that necessary to renumber these indices to something like 0 to 4? or is there any alternative? Dongyu ________________________________ From: Mark Adams Sent: Wednesday, July 3, 2019 3:54:49 PM To: Dongyu Liu - CITG Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to create a mapping of global to local indices? PETSc matrices and vectors are created with a local size n or global size N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from 0 to n_0 - 1 where n_0 is the number of equations on process 0. This numbering continues for all processes. You can use: PetscErrorCode MatGetOwnershipRange(Mat mat,PetscInt *m,PetscInt *n) To see what your local global indices are (rows m to n-1 are on this process and so its the local indices are 0 to (n-m-1)). Mark On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users > wrote: Hi, I am running a FEM program using PETSc. In the beginning, the mesh is partitioned in gmsh, and we read the partitioned mesh using our own reader. Now my question is: How can I get a global to local indices mapping? and do I need to renumber the indices after I read the partitioned mesh? >From my understanding, the rows handled by each processor should be consecutive, but the indices of nodes which I directly read from a gmsh file are not. Is this an issue? Best, Dongyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 3 09:26:47 2019 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 3 Jul 2019 10:26:47 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. Please give it a try. Thanks, Mark On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello everyone, > > When I run the ex19 with cuda like this: > mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor > -mat_type aijcusparse -vec_type cuda -log_view > > it worked fine and produced correct results. > > However, when I tried to run this example with nvprof: > nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 > -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda > -log_view > > I got errors like: > [3]PETSC ERROR: Error in external library > [3]PETSC ERROR: CUDA error 700 > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown > [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 > 2019 > [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 > --with-mpi-dir=/home/eps/MyLocal/mpi/mpich > --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl > --with-cuda-dir=/home/eps/MyLocal/cuda > --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install > --download-hdf5=1 > [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [3]PETSC ERROR: #2 VecSet() line 547 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c > [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #4 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in > /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #6 VecSetType() line 51 in > /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in > /home/eps/MyLocal/petsc/src/mat/interface/matrix.c > [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in > /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c > [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in > /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c > [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in > /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c > [3]PETSC ERROR: #12 SNESSolve() line 4560 in > /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > > The full run log is attached. > > I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA > Version: 10.1. > > I do not know why it is okay without nvprof but crashed with nvprof. Any > suggestion to fix this? > > Thank you. > > Best, > Xiangdong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amfoggia at gmail.com Wed Jul 3 09:37:14 2019 From: amfoggia at gmail.com (Ale Foggia) Date: Wed, 3 Jul 2019 16:37:14 +0200 Subject: [petsc-users] Communication during MatAssemblyEnd In-Reply-To: <4a837988-89a5-afce-6a93-c702e284753b@anl.gov> References: <845985ccc231460caea935e3e360c87c@DM6PR09MB3369.namprd09.prod.outlook.com> <4a837988-89a5-afce-6a93-c702e284753b@anl.gov> Message-ID: Thank you Richard for your explanation. I first changed the way I was running the code. In the machine there's SLURM and I was using "srun -n ./my_program.x". I've seen more than 60% improvement in execution time by just running with "srun -n --ntasks-per-core=1 --cpu-bind=cores> ./my_program.x". All the results I'm sharing now were obtained by running with the second command. 1) I've tested the configuration you suggested me and I've found mixed results. For one system (matrix) size and distribution of memory per node, there's a small difference in execution time (say, it goes from 208 seconds to 200 seconds) between the two cases: I_MPI_PIN_DOMAIN=socket and I_MPI_PIN_DOMAIN=auto:compact. In another case, I obtained that using I_MPI_PIN_DOMAIN=auto:socket gives actually worse results: I go from an execution time of 5 second to 11 seconds, when using "socket" and "auto:compact" configurations respectively. In this case, I see that the log events that differ in execution time are MatMult, VecNorm and VecScatterEnd. I also see that the imbalance for those log events is smaller when using "socket" (consequently the time difference). 2) I also tested using I_MPI_PIN_PROCESSOR_LIST=0-63 and I get the same numbers that I get when using I_MPI_PIN_DOMAIN=socket (I've seen the BIOS numbering and those correspond to the first of the CPUs in each core). El mar., 25 jun. 2019 a las 2:32, Mills, Richard Tran () escribi?: > Hi Ale, > > I don't know if this has anything to do with the strange performance you > are seeing, but I notice that some of your Intel MPI settings are > inconsistent and I'm not sure what you are intending. You have specified a > value for I_MPI_PIN_DOMAIN and also a value for I_MPI_PIN_PROCESSOR_LIST. > You can specify the domains that each MPI rank is pinned to by specifying > one of these, but not both. According to what I found by a Google search > for the Intel MPI documentation, if you specify both, it is > I_MPI_PIN_DOMAIN that gets used. > > A KNL node has only one socket per node (though it can be made to appear > as if it has four virtual sockets if booted into SNC-4 mode, which doesn't > seem to be a popular usage model). If you run with > "I_MPI_PIN_DOMAIN=socket", then each of your MPI ranks on a node will be > "pinned" to a domain that consists of all of the CPU cores available in a > socket, so the MPI processes aren't really pinned at all, and can be > migrated around by the system. You may see better performance by having > your MPI implementation pin processes to more restrictive domains. I > suggest first trying > > I_MPI_PIN_DOMAIN=auto:compact > > which I believe is the default Intel MPI behavior. This will create > domains by dividing the number of logical CPUs (# of hardware threads) by > the number of MPI processes being started on the node, and they will be > "compact" in the sense that domain members will be as "close" to each other > as possible in terms of sharing cores, caches, etc. > > I think that Intel MPI is ignoring your I_MPI_PIN_PROCESSOR_LIST setting, > but I'll go ahead and point out that, if you are running with 64-core KNL > nodes, there are 4*64 = 256 logical processors available, because each core > supports four hardware threads. Asking for logical processors 0-63 might > not actually be using all of the cores, as, depending on the BIOS numbering > (which can be arbitrary), some of these logical processors may actually be > hardware threads that share the same core. > > Best regards, > Richard > > On 6/24/19 4:19 AM, Ale Foggia via petsc-users wrote: > > Yes, I used KNL nodes. I you can perform the test would be great. Could it > be that I'm not using the correct configuration of the KNL nodes? These are > the environment variables I set: > MKL_NUM_THREADS=1 > OMP_NUM_THREADS=1 > KMP_HW_SUBSET=1t > KMP_AFFINITY=compact > I_MPI_PIN_DOMAIN=socket > I_MPI_PIN_PROCESSOR_LIST=0-63 > MKL_DYNAMIC=0 > > The code is in https://github.com/amfoggia/LSQuantumED and it has a > readme to compile it and run it. When I run the test I used only 32 > processors per node, and I used 1024 nodes in total, and it's for nspins=38. > Thank you > > El vie., 21 jun. 2019 a las 20:03, Zhang, Junchao () > escribi?: > >> Ale, >> Did you use Intel KNL nodes? Mr. Hong (cc'ed) did experiments on KNL >> nodes one year ago. He used 32768 processors and called MatAssemblyEnd 118 >> times and it used only 1.5 seconds in total. So I guess something was >> wrong with your test. If you can share your code, I can have a test on our >> machine to see how it goes. >> Thanks. >> --Junchao Zhang >> >> >> On Fri, Jun 21, 2019 at 11:00 AM Junchao Zhang >> wrote: >> >>> MatAssembly was called once (in stage 5) and cost 2.5% of the total >>> time. Look at stage 5. It says MatAssemblyBegin calls BuildTwoSidedF, >>> which does global synchronization. The high max/min ratio means load >>> imbalance. What I do not understand is MatAssemblyEnd. The ratio is 1.0. It >>> means processors are already synchronized. With 32768 processors, there are >>> 1.2e+06 messages with average length 1.9e+06 bytes. So each processor sends >>> 36 (1.2e+06/32768) ~2MB messages and it takes 54 seconds. Another chance is >>> the reduction at MatAssemblyEnd. I don't know why it needs 8 reductions. >>> In my mind, one is enough. I need to look at the code. >>> >>> Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages >>> --- -- Message Lengths -- -- Reductions -- >>> Avg %Total Avg %Total Count >>> %Total Avg %Total Count %Total >>> 0: Main Stage: 8.5045e+02 13.0% 3.0633e+15 14.0% 8.196e+07 >>> 13.1% 7.768e+06 13.1% 2.530e+02 13.0% >>> 1: Create Basis: 7.9234e-02 0.0% 0.0000e+00 0.0% 0.000e+00 >>> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>> 2: Create Lattice: 8.3944e-05 0.0% 0.0000e+00 0.0% 0.000e+00 >>> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>> 3: Create Hamilt: 1.0694e+02 1.6% 0.0000e+00 0.0% 0.000e+00 >>> 0.0% 0.000e+00 0.0% 2.000e+00 0.1% >>> 5: Offdiag: 1.6525e+02 2.5% 0.0000e+00 0.0% 1.188e+06 >>> 0.2% 1.942e+06 0.0% 8.000e+00 0.4% >>> 6: Phys quantities: 5.4045e+03 82.8% 1.8866e+16 86.0% 5.417e+08 >>> 86.7% 7.768e+06 86.8% 1.674e+03 86.1% >>> >>> --- Event Stage 5: Offdiag >>> BuildTwoSidedF 1 1.0 7.1565e+01 148448.9 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >>> MatAssemblyBegin 1 1.0 7.1565e+01 127783.7 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 1 0 0 0 0 28 0 0 0 0 0 >>> MatAssemblyEnd 1 1.0 5.3762e+01 1.0 0.00e+00 0.0 1.2e+06 >>> 1.9e+06 8.0e+00 1 0 0 0 0 33 0100100100 0 >>> VecSet 1 1.0 7.5533e-02 9.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jun 21, 2019 at 10:34 AM Smith, Barry F. >>> wrote: >>> >>>> >>>> The load balance is definitely out of whack. >>>> >>>> >>>> >>>> BuildTwoSidedF 1 1.0 1.6722e-0241.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatMult 138 1.0 2.6604e+02 7.4 3.19e+10 2.1 8.2e+07 >>>> 7.8e+06 0.0e+00 2 4 13 13 0 15 25100100 0 2935476 >>>> MatAssemblyBegin 1 1.0 1.6807e-0236.1 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyEnd 1 1.0 3.5680e-01 3.9 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecNorm 2 1.0 4.4252e+0174.8 1.73e+07 1.0 0.0e+00 >>>> 0.0e+00 2.0e+00 1 0 0 0 0 5 0 0 0 1 12780 >>>> VecCopy 6 1.0 6.5655e-02 2.6 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAXPY 2 1.0 1.3793e-02 2.7 1.73e+07 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41000838 >>>> VecScatterBegin 138 1.0 1.1653e+0285.8 0.00e+00 0.0 8.2e+07 >>>> 7.8e+06 0.0e+00 1 0 13 13 0 4 0100100 0 0 >>>> VecScatterEnd 138 1.0 1.3653e+0222.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 >>>> VecSetRandom 1 1.0 9.6668e-01 2.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> >>>> Note that VecCopy/AXPY/SetRandom which are all embarrassingly parallel >>>> have a balance ratio above 2 which means some processes have more than >>>> twice the work of others. Meanwhile the ratio for anything with >>>> communication is extremely in balanced, some processes get to the >>>> synchronization point well before other processes. >>>> >>>> The first thing I would do is worry about the load imbalance, what is >>>> its cause? is it one process with much less work than others (not great but >>>> not terrible) or is it one process with much more work then the others >>>> (terrible) or something in between. I think once you get a handle on the >>>> load balance the rest may fall into place, otherwise we still have some >>>> exploring to do. This is not expected behavior for a good machine with a >>>> good network and a well balanced job. After you understand the load >>>> balancing you may need to use one of the parallel performance visualization >>>> tools to see why the synchronization is out of whack. >>>> >>>> Good luck >>>> >>>> Barry >>>> >>>> >>>> > On Jun 21, 2019, at 9:27 AM, Ale Foggia wrote: >>>> > >>>> > I'm sending one with a bit less time. >>>> > I'm timing the functions also with std::chronos and for the case of >>>> 180 seconds the program runs out of memory (and crushes) before the PETSc >>>> log gets to be printed, so I know the time only from my function. Anyway, >>>> in every case, the times between std::chronos and the PETSc log match. >>>> > >>>> > (The large times are in part "4b- Building offdiagonal part" or >>>> "Event Stage 5: Offdiag"). >>>> > >>>> > El vie., 21 jun. 2019 a las 16:09, Zhang, Junchao (< >>>> jczhang at mcs.anl.gov>) escribi?: >>>> > >>>> > >>>> > On Fri, Jun 21, 2019 at 8:07 AM Ale Foggia >>>> wrote: >>>> > Thanks both of you for your answers, >>>> > >>>> > El jue., 20 jun. 2019 a las 22:20, Smith, Barry F. (< >>>> bsmith at mcs.anl.gov>) escribi?: >>>> > >>>> > Note that this is a one time cost if the nonzero structure of the >>>> matrix stays the same. It will not happen in future MatAssemblies. >>>> > >>>> > > On Jun 20, 2019, at 3:16 PM, Zhang, Junchao via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> > > >>>> > > Those messages were used to build MatMult communication pattern for >>>> the matrix. They were not part of the matrix entries-passing you imagined, >>>> but indeed happened in MatAssemblyEnd. If you want to make sure processors >>>> do not set remote entries, you can use >>>> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE), which will generate an >>>> error when an off-proc entry is set. >>>> > >>>> > I started being concerned about this when I saw that the assembly was >>>> taking a few hundreds of seconds in my code, like 180 seconds, which for me >>>> it's a considerable time. Do you think (or maybe you need more information >>>> to answer this) that this time is "reasonable" for communicating the >>>> pattern for the matrix? I already checked that I'm not setting any remote >>>> entries. >>>> > It is not reasonable. Could you send log view of that test with 180 >>>> seconds MatAssembly? >>>> > >>>> > Also I see (in my code) that even if there are no messages being >>>> passed during MatAssemblyBegin, it is taking time and the "ratio" is very >>>> big. >>>> > >>>> > > >>>> > > >>>> > > --Junchao Zhang >>>> > > >>>> > > >>>> > > On Thu, Jun 20, 2019 at 4:13 AM Ale Foggia via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> > > Hello all! >>>> > > >>>> > > During the conference I showed you a problem happening during >>>> MatAssemblyEnd in a particular code that I have. Now, I tried the same with >>>> a simple code (a symmetric problem corresponding to the Laplacian operator >>>> in 1D, from the SLEPc Hands-On exercises). As I understand (and please, >>>> correct me if I'm wrong), in this case the elements of the matrix are >>>> computed locally by each process so there should not be any communication >>>> during the assembly. However, in the log I get that there are messages >>>> being passed. Also, the number of messages changes with the number of >>>> processes used and the size of the matrix. Could you please help me >>>> understand this? >>>> > > >>>> > > I attach the code I used and the log I get for a small problem. >>>> > > >>>> > > Cheers, >>>> > > Ale >>>> > > >>>> > >>>> > >>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 3 12:55:28 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Jul 2019 12:55:28 -0500 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: <963a6ee9947046a6909192b369cd44b4@tudelft.nl> References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> <963a6ee9947046a6909192b369cd44b4@tudelft.nl> Message-ID: On Wed, Jul 3, 2019 at 9:07 AM Dongyu Liu - CITG via petsc-users < petsc-users at mcs.anl.gov> wrote: > Thank you, Mark. My question is if I have some node indices like [0, 1, 3, > 8, 9] handled by the current processor, is that necessary to renumber these > indices to something like 0 to 4? > Yes. > or is there any alternative? > No. Thanks, Matt > > Dongyu > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, July 3, 2019 3:54:49 PM > *To:* Dongyu Liu - CITG > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] How to create a mapping of global to local > indices? > > PETSc matrices and vectors are created with a local size n or global size > N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from > 0 to n_0 - 1 where n_0 is the number of equations on process 0. This > numbering continues for all processes. > > You can use: > > PetscErrorCode MatGetOwnershipRange (Mat mat,PetscInt *m,PetscInt *n) > > To see what your local global indices are (rows m to n-1 are on this > process and so its the local indices are 0 to (n-m-1)). > > Mark > > > On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi, >> >> >> I am running a FEM program using PETSc. In the beginning, the mesh is >> partitioned in gmsh, and we read the partitioned mesh using our own reader. >> >> >> Now my question is: How can I get a global to local indices mapping? and >> do I need to renumber the indices after I read the partitioned mesh? >> >> >> From my understanding, the rows handled by each processor should be >> consecutive, but the indices of nodes which I directly read from a gmsh >> file are not. Is this an issue? >> >> >> Best, >> >> >> Dongyu >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Wed Jul 3 13:03:42 2019 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Wed, 3 Jul 2019 20:03:42 +0200 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: References: Message-ID: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> Found the change... it was in Commit 6d563f2afc7bcefff9dc7869193a6debf1d639a5 Date: Wed Jun 26 12:32:01 2019 +0000 remove remnants of non-existing DMPlexRestore{Cone,ConeOrientation,Support} The patch removed the fortran stubs but not the C functions. In this case I guess that the fortran compiler reverts to the default calling convention and thus does not nullify the cone pointers but rather nulls the values in the cone. I suggest you remove the C functions to get a compiler error or revert the path. Many thanks, Fabian On 7/3/19 3:18 PM, Matthew Knepley wrote: > On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Matt, Dear Petsc >> >> After my recent pull of petsc master I encounter test fails that use >> dmplex meshes. >> >> I attached a simple script to show an issue where DMPlexRestoreCone >> mangles the cone relationships in the DM. >> >> The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 >> @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 >> >> We recently had a system update and I am not 100% sure if this is some >> weird stuff going on at my end or if this is indeed Plex related or if I >> should not call Restore in the inner loop. >> >> Anyway, I would be most grateful if you could try to reproduce the error >> and hint me towards a solution. >> > > Hi Fabian, > > I think it might be your system update. Seems to run fine for me: > > master *:~/Downloads/tmp$ ./plex_prism > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > > Thanks, > > > Matt > > >> As always, many many thanks! >> >> Fabian >> >> P.S. here the output of >> make run >> >> and note the cones, the second time they are outputted... >> >> ./plex_prism -show_plex ::ascii_info_detail >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 1 (0) >> [0]: 0 <---- 2 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 6 (0) >> [0]: 1 <---- 7 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 12 (0) >> [0]: 2 <---- 13 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 7 (0) >> [0]: 3 <---- 9 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 6 (0) >> [0]: 4 <---- 9 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 8 (0) >> [0]: 5 <---- 10 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 15 (0) >> [0]: 6 <---- 16 (0) >> [0]: 7 <---- 15 (0) >> [0]: 7 <---- 17 (0) >> [0]: 8 <---- 16 (0) >> [0]: 8 <---- 17 (0) >> [0]: 9 <---- 15 (0) >> [0]: 9 <---- 18 (0) >> [0]: 10 <---- 16 (0) >> [0]: 10 <---- 19 (0) >> [0]: 11 <---- 17 (0) >> [0]: 11 <---- 20 (0) >> [0]: 12 <---- 18 (0) >> [0]: 12 <---- 19 (0) >> [0]: 13 <---- 18 (0) >> [0]: 13 <---- 20 (0) >> [0]: 14 <---- 19 (0) >> [0]: 14 <---- 20 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> >> >> >> >> >> >> >> >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 0 (0) >> [0]: 6 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> > From bsmith at mcs.anl.gov Wed Jul 3 14:02:21 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 3 Jul 2019 19:02:21 +0000 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> <963a6ee9947046a6909192b369cd44b4@tudelft.nl> Message-ID: Note that number ingof vertices/nodes whatever is just an arbitrary label; no one is better or more fundamental than another. In your code's interaction with PETSc vectors and matrices you must use the contiguous numbering, in your mesh management code you can do whatever you want. There are routines AOCreate(), AOApplicationToPetsc() etc that you can, if you choose, use to renumber the vertices/nodes in your mesh data structure to match those used by PETSc vectors/matrices. Barry > On Jul 3, 2019, at 12:55 PM, Matthew Knepley via petsc-users wrote: > > On Wed, Jul 3, 2019 at 9:07 AM Dongyu Liu - CITG via petsc-users wrote: > Thank you, Mark. My question is if I have some node indices like [0, 1, 3, 8, 9] handled by the current processor, is that necessary to renumber these indices to something like 0 to 4? > > Yes. > > or is there any alternative? > > No. > > Thanks, > > Matt > > > Dongyu > > From: Mark Adams > Sent: Wednesday, July 3, 2019 3:54:49 PM > To: Dongyu Liu - CITG > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to create a mapping of global to local indices? > > PETSc matrices and vectors are created with a local size n or global size N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from 0 to n_0 - 1 where n_0 is the number of equations on process 0. This numbering continues for all processes. > > You can use: > PetscErrorCode MatGetOwnershipRange(Mat mat,PetscInt *m,PetscInt *n) > To see what your local global indices are (rows m to n-1 are on this process and so its the local indices are 0 to (n-m-1)). > > Mark > > > On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users wrote: > Hi, > > > > I am running a FEM program using PETSc. In the beginning, the mesh is partitioned in gmsh, and we read the partitioned mesh using our own reader. > > > > Now my question is: How can I get a global to local indices mapping? and do I need to renumber the indices after I read the partitioned mesh? > > > > From my understanding, the rows handled by each processor should be consecutive, but the indices of nodes which I directly read from a gmsh file are not. Is this an issue? > > > > Best, > > > > Dongyu > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From mvalera-w at sdsu.edu Wed Jul 3 15:10:38 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 3 Jul 2019 13:10:38 -0700 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: Message-ID: Thanks Zhang for your answer, I ended up getting a compiling and running TS routine... that does not give me the answers, and i am not sure what i am doing wrong, My TS code so far looks like this: (...initialization...) call TSCreate(PETSC_COMM_WORLD,ts,ierr) call TSSetProblemType(ts,TS_NONLINEAR,ierr) call TSSetSolution(ts,gTemperature,ierr) call TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) call TSSetType(ts,TSSSP,ierr) call TSSetTimeStep(ts,dt,ierr) call TSSetDM(ts,daScalars,ierr) call TSSetMaxTime(ts,TotalTime,ierr) call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) call TSSetFromOptions(ts,ierr) call TSSetUp(ts,ierr) (... then inside main calculations loop i have...) call TSSolve(ts,gTemperature,ierr) (...and my RHSFunction looks like this...) subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) use petscvec use petscts use petscdmda use petscdm USE utils USE dmdaobjs USE probsize USE modelparams implicit none TS :: ts PetscReal :: t Vec :: ctx,t_loc,rhs_loc PetscErrorCode :: ierr call TSGetDM(ts,daScalars,ierr) call DMGetLocalVector(daScalars,t_loc,ierr); call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) call DMDASolveTExplicit(3) call DMGetLocalVector(daScalars,rhs_loc,ierr); call DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) call DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) end subroutine FormRHSFunction Where DMDASolveTExplicit(3) is the old function to calculate time integration with runge kutta, modified to only generate the f(t,u) which in this case is rhs_loc, I still have several doubts: - Will this explicit RHS calculation work with TS routines? my time integration is explicit so far and it would involve a fair deal of extra work to make it implicit. - This 't' parameter in the RHSFunction is controlled by PETSC? i am not providing anything for it directly, where is it coming from? - Do i need to provide a Jacobian or the TS routine will try to guess one? This is related to the first point where my time scheme being explicit does not use a jacobian. Thanks, any help is appreciated, if you see any errors or need more information please let me know, Regards, Manuel On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: > > > On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hi PETSc, > > I am trying to implement the Time stepping routines in my model, i have a > working runge-kutta time scheme that goes to the following steps: > > > - Interpolate u,v,w to the time advancing variable position. > - Calculate nonlinear coefficients and advect velocities with a > forward-backward shock capturing scheme. > - Calculate the variable laplacian > - Sum terms to create RHS (nonlinear advected velocities + laplacian) > - Finally, the runge kutta integration is done in a typical way that > looks like: > > newtemp(t+1) = prevtemp(t) + dt*RHS > > > So my questions are: > > - I think my problem is nonlinear, but is being made linearized by the > advecting scheme, is this correct? this is to know if i should use the > linear or nonlinear routines for TS. > > TSComputeRHSFunctionLinear is just a convenience function for linear ODEs > in the form udot=Au. Using it won?t buy you much. So for TS starters, it is > fine to assume your problem is nonlinear and think of the form udot=f(t,u) > where f is the RHS function. > > > - How do i know what are the appropriate routines i should be using > here? so far i think i should use the following: > > call TSCreate(PETSC_COMM_WORLD,ts,ierr) > call TSSetProblemType(ts,TS_LINEAR,ierr) > call TSSetTimeStep(ts,dt,ierr) > > call TSSetFromOptions(ts,ierr) > > call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) > call TSSolve(ts,loctemperature,ierr) > > Should i use call TSSetRHSJacobian for the temperature jacobian in this > case? > > > I would suggest to write your own RHSFunction and set it to TS with > TSSetRHSFunction(). > > > > I am using > https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html as > a general guide, is there a more appropriate example? > > > ex4 is a good example. In addition, ex9 uses finite volume method with > slope limiters to solve a variety of problems such as advection equation, > burgers equation and shallow water equation. It might be an overkill, but > it seems to be close to the problem you are trying to solve. Note that you > might want to use the SSP methods (-ts_type ssp) instead of the classic > Runge-Kutta methods for problems with shocks. > > Hong (Mr.) > > > The dt value and total timesteps are controlled by the model, > > Thanks for your help, > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From D.Liu-4 at tudelft.nl Thu Jul 4 08:34:11 2019 From: D.Liu-4 at tudelft.nl (Dongyu Liu - CITG) Date: Thu, 4 Jul 2019 13:34:11 +0000 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> <963a6ee9947046a6909192b369cd44b4@tudelft.nl> , Message-ID: <0b20b45dae8e41928c7cccf4794c4761@tudelft.nl> Thanks, Smith. I understand the functionalities of the AOCreate, AOApplicationToPetsc(), they can be used to basicially convert old global indices to new indices. But my next question will be how can they cooperate with the sparse matrix? As I understand, these objects behave kind of like maps, but I don't know how to pass this information when I create the global stiffness matrix or add value to the global stiffness matrix. Best, Dongyu ________________________________ From: Smith, Barry F. Sent: Wednesday, July 3, 2019 9:02:21 PM To: Matthew Knepley Cc: Dongyu Liu - CITG; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to create a mapping of global to local indices? Note that number ingof vertices/nodes whatever is just an arbitrary label; no one is better or more fundamental than another. In your code's interaction with PETSc vectors and matrices you must use the contiguous numbering, in your mesh management code you can do whatever you want. There are routines AOCreate(), AOApplicationToPetsc() etc that you can, if you choose, use to renumber the vertices/nodes in your mesh data structure to match those used by PETSc vectors/matrices. Barry > On Jul 3, 2019, at 12:55 PM, Matthew Knepley via petsc-users wrote: > > On Wed, Jul 3, 2019 at 9:07 AM Dongyu Liu - CITG via petsc-users wrote: > Thank you, Mark. My question is if I have some node indices like [0, 1, 3, 8, 9] handled by the current processor, is that necessary to renumber these indices to something like 0 to 4? > > Yes. > > or is there any alternative? > > No. > > Thanks, > > Matt > > > Dongyu > > From: Mark Adams > Sent: Wednesday, July 3, 2019 3:54:49 PM > To: Dongyu Liu - CITG > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to create a mapping of global to local indices? > > PETSc matrices and vectors are created with a local size n or global size N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from 0 to n_0 - 1 where n_0 is the number of equations on process 0. This numbering continues for all processes. > > You can use: > PetscErrorCode MatGetOwnershipRange(Mat mat,PetscInt *m,PetscInt *n) > To see what your local global indices are (rows m to n-1 are on this process and so its the local indices are 0 to (n-m-1)). > > Mark > > > On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users wrote: > Hi, > > > > I am running a FEM program using PETSc. In the beginning, the mesh is partitioned in gmsh, and we read the partitioned mesh using our own reader. > > > > Now my question is: How can I get a global to local indices mapping? and do I need to renumber the indices after I read the partitioned mesh? > > > > From my understanding, the rows handled by each processor should be consecutive, but the indices of nodes which I directly read from a gmsh file are not. Is this an issue? > > > > Best, > > > > Dongyu > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 4 08:36:36 2019 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jul 2019 08:36:36 -0500 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> References: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> Message-ID: On Wed, Jul 3, 2019 at 1:03 PM Fabian.Jakub < Fabian.Jakub at physik.uni-muenchen.de> wrote: > Found the change... > it was in Commit 6d563f2afc7bcefff9dc7869193a6debf1d639a5 > Date: Wed Jun 26 12:32:01 2019 +0000 > > remove remnants of non-existing > DMPlexRestore{Cone,ConeOrientation,Support} > You are right! Vaclav, you cannot remove the Fortran RestoreCone() because it is actually doing something. Fortran needs to create arrays, and this code deallocates them. We do not need that in C. Can you put those back? Thanks, Matt > The patch removed the fortran stubs but not the C functions. > In this case I guess that the fortran compiler reverts to the default > calling convention and thus does not nullify the cone pointers but > rather nulls the values in the cone. > > I suggest you remove the C functions to get a compiler error or revert > the path. > > Many thanks, > > Fabian > > On 7/3/19 3:18 PM, Matthew Knepley wrote: > > On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > >> Dear Matt, Dear Petsc > >> > >> After my recent pull of petsc master I encounter test fails that use > >> dmplex meshes. > >> > >> I attached a simple script to show an issue where DMPlexRestoreCone > >> mangles the cone relationships in the DM. > >> > >> The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 > >> @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 > >> > >> We recently had a system update and I am not 100% sure if this is some > >> weird stuff going on at my end or if this is indeed Plex related or if I > >> should not call Restore in the inner loop. > >> > >> Anyway, I would be most grateful if you could try to reproduce the error > >> and hint me towards a solution. > >> > > > > Hi Fabian, > > > > I think it might be your system update. Seems to run fine for me: > > > > master *:~/Downloads/tmp$ ./plex_prism > > > > Cone 0 : 1 2 3 4 > > 5 > > > > Cone 1 : 6 7 8 > > > > Cone 2 : 12 13 14 > > > > Cone 3 : 7 9 11 13 > > > > Cone 4 : 6 9 10 12 > > > > Cone 5 : 8 10 11 14 > > > > Cone 6 : 15 16 > > > > Cone 7 : 15 17 > > > > Cone 8 : 16 17 > > > > Cone 9 : 15 18 > > > > Cone 10 : 16 19 > > > > Cone 11 : 17 20 > > > > Cone 12 : 18 19 > > > > Cone 13 : 18 20 > > > > Cone 14 : 19 20 > > > > Cone 15 : > > > > Cone 16 : > > > > Cone 17 : > > > > Cone 18 : > > > > Cone 19 : > > > > Cone 20 : > > > > master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail > > > > DM Object: testplex 1 MPI processes > > > > type: plex > > > > testplex in 3 dimensions: > > > > Supports: > > > > [0] Max support size: 3 > > > > [0]: 1 ----> 0 > > > > [0]: 2 ----> 0 > > > > [0]: 3 ----> 0 > > > > [0]: 4 ----> 0 > > > > [0]: 5 ----> 0 > > > > [0]: 6 ----> 1 > > > > [0]: 6 ----> 4 > > > > [0]: 7 ----> 1 > > > > [0]: 7 ----> 3 > > > > [0]: 8 ----> 1 > > > > [0]: 8 ----> 5 > > > > [0]: 9 ----> 3 > > > > [0]: 9 ----> 4 > > > > [0]: 10 ----> 4 > > > > [0]: 10 ----> 5 > > > > [0]: 11 ----> 3 > > > > [0]: 11 ----> 5 > > > > [0]: 12 ----> 2 > > > > [0]: 12 ----> 4 > > > > [0]: 13 ----> 2 > > > > [0]: 13 ----> 3 > > > > [0]: 14 ----> 2 > > > > [0]: 14 ----> 5 > > > > [0]: 15 ----> 6 > > > > [0]: 15 ----> 7 > > > > [0]: 15 ----> 9 > > > > [0]: 16 ----> 6 > > > > [0]: 16 ----> 8 > > > > [0]: 16 ----> 10 > > > > [0]: 17 ----> 7 > > > > [0]: 17 ----> 8 > > > > [0]: 17 ----> 11 > > > > [0]: 18 ----> 9 > > > > [0]: 18 ----> 12 > > > > [0]: 18 ----> 13 > > > > [0]: 19 ----> 10 > > > > [0]: 19 ----> 12 > > > > [0]: 19 ----> 14 > > > > [0]: 20 ----> 11 > > > > [0]: 20 ----> 13 > > > > [0]: 20 ----> 14 > > > > Cones: > > > > [0] Max cone size: 5 > > > > [0]: 0 <---- 1 (0) > > > > [0]: 0 <---- 2 (0) > > > > [0]: 0 <---- 3 (0) > > > > [0]: 0 <---- 4 (0) > > > > [0]: 0 <---- 5 (0) > > > > [0]: 1 <---- 6 (0) > > > > [0]: 1 <---- 7 (0) > > > > [0]: 1 <---- 8 (0) > > > > [0]: 2 <---- 12 (0) > > > > [0]: 2 <---- 13 (0) > > > > [0]: 2 <---- 14 (0) > > > > [0]: 3 <---- 7 (0) > > > > [0]: 3 <---- 9 (0) > > > > [0]: 3 <---- 11 (0) > > > > [0]: 3 <---- 13 (0) > > > > [0]: 4 <---- 6 (0) > > > > [0]: 4 <---- 9 (0) > > > > [0]: 4 <---- 10 (0) > > > > [0]: 4 <---- 12 (0) > > > > [0]: 5 <---- 8 (0) > > > > [0]: 5 <---- 10 (0) > > > > [0]: 5 <---- 11 (0) > > > > [0]: 5 <---- 14 (0) > > > > [0]: 6 <---- 15 (0) > > > > [0]: 6 <---- 16 (0) > > > > [0]: 7 <---- 15 (0) > > > > [0]: 7 <---- 17 (0) > > > > [0]: 8 <---- 16 (0) > > > > [0]: 8 <---- 17 (0) > > > > [0]: 9 <---- 15 (0) > > > > [0]: 9 <---- 18 (0) > > > > [0]: 10 <---- 16 (0) > > > > [0]: 10 <---- 19 (0) > > > > [0]: 11 <---- 17 (0) > > > > [0]: 11 <---- 20 (0) > > > > [0]: 12 <---- 18 (0) > > > > [0]: 12 <---- 19 (0) > > > > [0]: 13 <---- 18 (0) > > > > [0]: 13 <---- 20 (0) > > > > [0]: 14 <---- 19 (0) > > > > [0]: 14 <---- 20 (0) > > > > coordinates with 1 fields > > > > field 0 with 3 components > > > > Process 0: > > > > ( 0) dim 0 offset 0 > > > > ( 1) dim 0 offset 0 > > > > ( 2) dim 0 offset 0 > > > > ( 3) dim 0 offset 0 > > > > ( 4) dim 0 offset 0 > > > > ( 5) dim 0 offset 0 > > > > ( 6) dim 0 offset 0 > > > > ( 7) dim 0 offset 0 > > > > ( 8) dim 0 offset 0 > > > > ( 9) dim 0 offset 0 > > > > ( 10) dim 0 offset 0 > > > > ( 11) dim 0 offset 0 > > > > ( 12) dim 0 offset 0 > > > > ( 13) dim 0 offset 0 > > > > ( 14) dim 0 offset 0 > > > > ( 15) dim 3 offset 0 0. 0. 0. > > > > ( 16) dim 3 offset 3 0. 2. 0. > > > > ( 17) dim 3 offset 6 2. 1. 0. > > > > ( 18) dim 3 offset 9 0. 0. 1. > > > > ( 19) dim 3 offset 12 0. 2. 1. > > > > ( 20) dim 3 offset 15 2. 1. 1. > > > > Cone 0 : 1 2 3 4 > > 5 > > > > Cone 1 : 6 7 8 > > > > Cone 2 : 12 13 14 > > > > Cone 3 : 7 9 11 13 > > > > Cone 4 : 6 9 10 12 > > > > Cone 5 : 8 10 11 14 > > > > Cone 6 : 15 16 > > > > Cone 7 : 15 17 > > > > Cone 8 : 16 17 > > > > Cone 9 : 15 18 > > > > Cone 10 : 16 19 > > > > Cone 11 : 17 20 > > > > Cone 12 : 18 19 > > > > Cone 13 : 18 20 > > > > Cone 14 : 19 20 > > > > Cone 15 : > > > > Cone 16 : > > > > Cone 17 : > > > > Cone 18 : > > > > Cone 19 : > > > > Cone 20 : > > > > DM Object: testplex 1 MPI processes > > > > type: plex > > > > testplex in 3 dimensions: > > > > Supports: > > > > [0] Max support size: 3 > > > > [0]: 1 ----> 0 > > > > [0]: 2 ----> 0 > > > > [0]: 3 ----> 0 > > > > [0]: 4 ----> 0 > > > > [0]: 5 ----> 0 > > > > [0]: 6 ----> 1 > > > > [0]: 6 ----> 4 > > > > [0]: 7 ----> 1 > > > > [0]: 7 ----> 3 > > > > [0]: 8 ----> 1 > > > > [0]: 8 ----> 5 > > > > [0]: 9 ----> 3 > > > > [0]: 9 ----> 4 > > > > [0]: 10 ----> 4 > > > > [0]: 10 ----> 5 > > > > [0]: 11 ----> 3 > > > > [0]: 11 ----> 5 > > > > [0]: 12 ----> 2 > > > > [0]: 12 ----> 4 > > > > [0]: 13 ----> 2 > > > > [0]: 13 ----> 3 > > > > [0]: 14 ----> 2 > > > > [0]: 14 ----> 5 > > > > [0]: 15 ----> 6 > > > > [0]: 15 ----> 7 > > > > [0]: 15 ----> 9 > > > > [0]: 16 ----> 6 > > > > [0]: 16 ----> 8 > > > > [0]: 16 ----> 10 > > > > [0]: 17 ----> 7 > > > > [0]: 17 ----> 8 > > > > [0]: 17 ----> 11 > > > > [0]: 18 ----> 9 > > > > [0]: 18 ----> 12 > > > > [0]: 18 ----> 13 > > > > [0]: 19 ----> 10 > > > > [0]: 19 ----> 12 > > > > [0]: 19 ----> 14 > > > > [0]: 20 ----> 11 > > > > [0]: 20 ----> 13 > > > > [0]: 20 ----> 14 > > > > Cones: > > > > [0] Max cone size: 5 > > > > [0]: 0 <---- 1 (0) > > > > [0]: 0 <---- 2 (0) > > > > [0]: 0 <---- 3 (0) > > > > [0]: 0 <---- 4 (0) > > > > [0]: 0 <---- 5 (0) > > > > [0]: 1 <---- 6 (0) > > > > [0]: 1 <---- 7 (0) > > > > [0]: 1 <---- 8 (0) > > > > [0]: 2 <---- 12 (0) > > > > [0]: 2 <---- 13 (0) > > > > [0]: 2 <---- 14 (0) > > > > [0]: 3 <---- 7 (0) > > > > [0]: 3 <---- 9 (0) > > > > [0]: 3 <---- 11 (0) > > > > [0]: 3 <---- 13 (0) > > > > [0]: 4 <---- 6 (0) > > > > [0]: 4 <---- 9 (0) > > > > [0]: 4 <---- 10 (0) > > > > [0]: 4 <---- 12 (0) > > > > [0]: 5 <---- 8 (0) > > > > [0]: 5 <---- 10 (0) > > > > [0]: 5 <---- 11 (0) > > > > [0]: 5 <---- 14 (0) > > > > [0]: 6 <---- 15 (0) > > > > [0]: 6 <---- 16 (0) > > > > [0]: 7 <---- 15 (0) > > > > [0]: 7 <---- 17 (0) > > > > [0]: 8 <---- 16 (0) > > > > [0]: 8 <---- 17 (0) > > > > [0]: 9 <---- 15 (0) > > > > [0]: 9 <---- 18 (0) > > > > [0]: 10 <---- 16 (0) > > > > [0]: 10 <---- 19 (0) > > > > [0]: 11 <---- 17 (0) > > > > [0]: 11 <---- 20 (0) > > > > [0]: 12 <---- 18 (0) > > > > [0]: 12 <---- 19 (0) > > > > [0]: 13 <---- 18 (0) > > > > [0]: 13 <---- 20 (0) > > > > [0]: 14 <---- 19 (0) > > > > [0]: 14 <---- 20 (0) > > > > coordinates with 1 fields > > > > field 0 with 3 components > > > > Process 0: > > > > ( 0) dim 0 offset 0 > > > > ( 1) dim 0 offset 0 > > > > ( 2) dim 0 offset 0 > > > > ( 3) dim 0 offset 0 > > > > ( 4) dim 0 offset 0 > > > > ( 5) dim 0 offset 0 > > > > ( 6) dim 0 offset 0 > > > > ( 7) dim 0 offset 0 > > > > ( 8) dim 0 offset 0 > > > > ( 9) dim 0 offset 0 > > > > ( 10) dim 0 offset 0 > > > > ( 11) dim 0 offset 0 > > > > ( 12) dim 0 offset 0 > > > > ( 13) dim 0 offset 0 > > > > ( 14) dim 0 offset 0 > > > > ( 15) dim 3 offset 0 0. 0. 0. > > > > ( 16) dim 3 offset 3 0. 2. 0. > > > > ( 17) dim 3 offset 6 2. 1. 0. > > > > ( 18) dim 3 offset 9 0. 0. 1. > > > > ( 19) dim 3 offset 12 0. 2. 1. > > > > ( 20) dim 3 offset 15 2. 1. 1. > > > > > > Thanks, > > > > > > Matt > > > > > >> As always, many many thanks! > >> > >> Fabian > >> > >> P.S. here the output of > >> make run > >> > >> and note the cones, the second time they are outputted... > >> > >> ./plex_prism -show_plex ::ascii_info_detail > >> DM Object: testplex 1 MPI processes > >> type: plex > >> testplex in 3 dimensions: > >> Supports: > >> [0] Max support size: 3 > >> [0]: 1 ----> 0 > >> [0]: 2 ----> 0 > >> [0]: 3 ----> 0 > >> [0]: 4 ----> 0 > >> [0]: 5 ----> 0 > >> [0]: 6 ----> 1 > >> [0]: 6 ----> 4 > >> [0]: 7 ----> 1 > >> [0]: 7 ----> 3 > >> [0]: 8 ----> 1 > >> [0]: 8 ----> 5 > >> [0]: 9 ----> 3 > >> [0]: 9 ----> 4 > >> [0]: 10 ----> 4 > >> [0]: 10 ----> 5 > >> [0]: 11 ----> 3 > >> [0]: 11 ----> 5 > >> [0]: 12 ----> 2 > >> [0]: 12 ----> 4 > >> [0]: 13 ----> 2 > >> [0]: 13 ----> 3 > >> [0]: 14 ----> 2 > >> [0]: 14 ----> 5 > >> [0]: 15 ----> 6 > >> [0]: 15 ----> 7 > >> [0]: 15 ----> 9 > >> [0]: 16 ----> 6 > >> [0]: 16 ----> 8 > >> [0]: 16 ----> 10 > >> [0]: 17 ----> 7 > >> [0]: 17 ----> 8 > >> [0]: 17 ----> 11 > >> [0]: 18 ----> 9 > >> [0]: 18 ----> 12 > >> [0]: 18 ----> 13 > >> [0]: 19 ----> 10 > >> [0]: 19 ----> 12 > >> [0]: 19 ----> 14 > >> [0]: 20 ----> 11 > >> [0]: 20 ----> 13 > >> [0]: 20 ----> 14 > >> Cones: > >> [0] Max cone size: 5 > >> [0]: 0 <---- 1 (0) > >> [0]: 0 <---- 2 (0) > >> [0]: 0 <---- 3 (0) > >> [0]: 0 <---- 4 (0) > >> [0]: 0 <---- 5 (0) > >> [0]: 1 <---- 6 (0) > >> [0]: 1 <---- 7 (0) > >> [0]: 1 <---- 8 (0) > >> [0]: 2 <---- 12 (0) > >> [0]: 2 <---- 13 (0) > >> [0]: 2 <---- 14 (0) > >> [0]: 3 <---- 7 (0) > >> [0]: 3 <---- 9 (0) > >> [0]: 3 <---- 11 (0) > >> [0]: 3 <---- 13 (0) > >> [0]: 4 <---- 6 (0) > >> [0]: 4 <---- 9 (0) > >> [0]: 4 <---- 10 (0) > >> [0]: 4 <---- 12 (0) > >> [0]: 5 <---- 8 (0) > >> [0]: 5 <---- 10 (0) > >> [0]: 5 <---- 11 (0) > >> [0]: 5 <---- 14 (0) > >> [0]: 6 <---- 15 (0) > >> [0]: 6 <---- 16 (0) > >> [0]: 7 <---- 15 (0) > >> [0]: 7 <---- 17 (0) > >> [0]: 8 <---- 16 (0) > >> [0]: 8 <---- 17 (0) > >> [0]: 9 <---- 15 (0) > >> [0]: 9 <---- 18 (0) > >> [0]: 10 <---- 16 (0) > >> [0]: 10 <---- 19 (0) > >> [0]: 11 <---- 17 (0) > >> [0]: 11 <---- 20 (0) > >> [0]: 12 <---- 18 (0) > >> [0]: 12 <---- 19 (0) > >> [0]: 13 <---- 18 (0) > >> [0]: 13 <---- 20 (0) > >> [0]: 14 <---- 19 (0) > >> [0]: 14 <---- 20 (0) > >> coordinates with 1 fields > >> field 0 with 3 components > >> Process 0: > >> ( 0) dim 0 offset 0 > >> ( 1) dim 0 offset 0 > >> ( 2) dim 0 offset 0 > >> ( 3) dim 0 offset 0 > >> ( 4) dim 0 offset 0 > >> ( 5) dim 0 offset 0 > >> ( 6) dim 0 offset 0 > >> ( 7) dim 0 offset 0 > >> ( 8) dim 0 offset 0 > >> ( 9) dim 0 offset 0 > >> ( 10) dim 0 offset 0 > >> ( 11) dim 0 offset 0 > >> ( 12) dim 0 offset 0 > >> ( 13) dim 0 offset 0 > >> ( 14) dim 0 offset 0 > >> ( 15) dim 3 offset 0 0. 0. 0. > >> ( 16) dim 3 offset 3 0. 2. 0. > >> ( 17) dim 3 offset 6 2. 1. 0. > >> ( 18) dim 3 offset 9 0. 0. 1. > >> ( 19) dim 3 offset 12 0. 2. 1. > >> ( 20) dim 3 offset 15 2. 1. 1. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> DM Object: testplex 1 MPI processes > >> type: plex > >> testplex in 3 dimensions: > >> Supports: > >> [0] Max support size: 3 > >> [0]: 1 ----> 0 > >> [0]: 2 ----> 0 > >> [0]: 3 ----> 0 > >> [0]: 4 ----> 0 > >> [0]: 5 ----> 0 > >> [0]: 6 ----> 1 > >> [0]: 6 ----> 4 > >> [0]: 7 ----> 1 > >> [0]: 7 ----> 3 > >> [0]: 8 ----> 1 > >> [0]: 8 ----> 5 > >> [0]: 9 ----> 3 > >> [0]: 9 ----> 4 > >> [0]: 10 ----> 4 > >> [0]: 10 ----> 5 > >> [0]: 11 ----> 3 > >> [0]: 11 ----> 5 > >> [0]: 12 ----> 2 > >> [0]: 12 ----> 4 > >> [0]: 13 ----> 2 > >> [0]: 13 ----> 3 > >> [0]: 14 ----> 2 > >> [0]: 14 ----> 5 > >> [0]: 15 ----> 6 > >> [0]: 15 ----> 7 > >> [0]: 15 ----> 9 > >> [0]: 16 ----> 6 > >> [0]: 16 ----> 8 > >> [0]: 16 ----> 10 > >> [0]: 17 ----> 7 > >> [0]: 17 ----> 8 > >> [0]: 17 ----> 11 > >> [0]: 18 ----> 9 > >> [0]: 18 ----> 12 > >> [0]: 18 ----> 13 > >> [0]: 19 ----> 10 > >> [0]: 19 ----> 12 > >> [0]: 19 ----> 14 > >> [0]: 20 ----> 11 > >> [0]: 20 ----> 13 > >> [0]: 20 ----> 14 > >> Cones: > >> [0] Max cone size: 5 > >> [0]: 0 <---- 0 (0) > >> [0]: 0 <---- 0 (0) > >> [0]: 0 <---- 3 (0) > >> [0]: 0 <---- 4 (0) > >> [0]: 0 <---- 5 (0) > >> [0]: 1 <---- 0 (0) > >> [0]: 1 <---- 0 (0) > >> [0]: 1 <---- 8 (0) > >> [0]: 2 <---- 0 (0) > >> [0]: 2 <---- 0 (0) > >> [0]: 2 <---- 14 (0) > >> [0]: 3 <---- 0 (0) > >> [0]: 3 <---- 0 (0) > >> [0]: 3 <---- 11 (0) > >> [0]: 3 <---- 13 (0) > >> [0]: 4 <---- 0 (0) > >> [0]: 4 <---- 0 (0) > >> [0]: 4 <---- 10 (0) > >> [0]: 4 <---- 12 (0) > >> [0]: 5 <---- 0 (0) > >> [0]: 5 <---- 0 (0) > >> [0]: 5 <---- 11 (0) > >> [0]: 5 <---- 14 (0) > >> [0]: 6 <---- 0 (0) > >> [0]: 6 <---- 0 (0) > >> [0]: 7 <---- 0 (0) > >> [0]: 7 <---- 0 (0) > >> [0]: 8 <---- 0 (0) > >> [0]: 8 <---- 0 (0) > >> [0]: 9 <---- 0 (0) > >> [0]: 9 <---- 0 (0) > >> [0]: 10 <---- 0 (0) > >> [0]: 10 <---- 0 (0) > >> [0]: 11 <---- 0 (0) > >> [0]: 11 <---- 0 (0) > >> [0]: 12 <---- 0 (0) > >> [0]: 12 <---- 0 (0) > >> [0]: 13 <---- 0 (0) > >> [0]: 13 <---- 0 (0) > >> [0]: 14 <---- 0 (0) > >> [0]: 14 <---- 0 (0) > >> coordinates with 1 fields > >> field 0 with 3 components > >> Process 0: > >> ( 0) dim 0 offset 0 > >> ( 1) dim 0 offset 0 > >> ( 2) dim 0 offset 0 > >> ( 3) dim 0 offset 0 > >> ( 4) dim 0 offset 0 > >> ( 5) dim 0 offset 0 > >> ( 6) dim 0 offset 0 > >> ( 7) dim 0 offset 0 > >> ( 8) dim 0 offset 0 > >> ( 9) dim 0 offset 0 > >> ( 10) dim 0 offset 0 > >> ( 11) dim 0 offset 0 > >> ( 12) dim 0 offset 0 > >> ( 13) dim 0 offset 0 > >> ( 14) dim 0 offset 0 > >> ( 15) dim 3 offset 0 0. 0. 0. > >> ( 16) dim 3 offset 3 0. 2. 0. > >> ( 17) dim 3 offset 6 2. 1. 0. > >> ( 18) dim 3 offset 9 0. 0. 1. > >> ( 19) dim 3 offset 12 0. 2. 1. > >> ( 20) dim 3 offset 15 2. 1. 1. > >> > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 4 10:45:32 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 4 Jul 2019 15:45:32 +0000 Subject: [petsc-users] How to create a mapping of global to local indices? In-Reply-To: <0b20b45dae8e41928c7cccf4794c4761@tudelft.nl> References: <0afc0a441f1b4a02ade56a866d41a34d@tudelft.nl> <963a6ee9947046a6909192b369cd44b4@tudelft.nl> <0b20b45dae8e41928c7cccf4794c4761@tudelft.nl> Message-ID: <0E3159AF-F9D2-4EA3-A711-579DD81AA3E3@mcs.anl.gov> You do not use the AO when adding values to the matrices or vectors. You use the AO to change the numbering inside your mesh data structure. Your mesh data structure then has contiguous number of degrees of freedom per process just like the Mat and Vec need so you use these values directly to put values into the matrix/vectors. For example after applying the AO your cell (triangular for simplicity) may have vertices 0, 22, 88 you would form the element stiffness can call MatSetValues(A,3,{0,22,38},3,{0,22,38},values, ADD_VALUES). Barry > On Jul 4, 2019, at 8:34 AM, Dongyu Liu - CITG wrote: > > Thanks, Smith. I understand the functionalities of the AOCreate, AOApplicationToPetsc(), they can be used to basicially convert old global indices to new indices. But my next question will be how can they cooperate with the sparse matrix? As I understand, these objects behave kind of like maps, but I don't know how to pass this information when I create the global stiffness matrix or add value to the global stiffness matrix. > > Best, > Dongyu > From: Smith, Barry F. > Sent: Wednesday, July 3, 2019 9:02:21 PM > To: Matthew Knepley > Cc: Dongyu Liu - CITG; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to create a mapping of global to local indices? > > > Note that number ingof vertices/nodes whatever is just an arbitrary label; no one is better or more fundamental than another. In your code's interaction with PETSc vectors and matrices you must use the contiguous numbering, in your mesh management code you can do whatever you want. > > There are routines AOCreate(), AOApplicationToPetsc() etc that you can, if you choose, use to renumber the vertices/nodes in your mesh data structure to match those used by PETSc vectors/matrices. > > > Barry > > > > On Jul 3, 2019, at 12:55 PM, Matthew Knepley via petsc-users wrote: > > > > On Wed, Jul 3, 2019 at 9:07 AM Dongyu Liu - CITG via petsc-users wrote: > > Thank you, Mark. My question is if I have some node indices like [0, 1, 3, 8, 9] handled by the current processor, is that necessary to renumber these indices to something like 0 to 4? > > > > Yes. > > > > or is there any alternative? > > > > No. > > > > Thanks, > > > > Matt > > > > > > Dongyu > > > > From: Mark Adams > > Sent: Wednesday, July 3, 2019 3:54:49 PM > > To: Dongyu Liu - CITG > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] How to create a mapping of global to local indices? > > > > PETSc matrices and vectors are created with a local size n or global size N and PETSC_DECIDE instead of n. The global PETSc indices are ordered from 0 to n_0 - 1 where n_0 is the number of equations on process 0. This numbering continues for all processes. > > > > You can use: > > PetscErrorCode MatGetOwnershipRange(Mat mat,PetscInt *m,PetscInt *n) > > To see what your local global indices are (rows m to n-1 are on this process and so its the local indices are 0 to (n-m-1)). > > > > Mark > > > > > > On Wed, Jul 3, 2019 at 9:11 AM Dongyu Liu - CITG via petsc-users wrote: > > Hi, > > > > > > > > I am running a FEM program using PETSc. In the beginning, the mesh is partitioned in gmsh, and we read the partitioned mesh using our own reader. > > > > > > > > Now my question is: How can I get a global to local indices mapping? and do I need to renumber the indices after I read the partitioned mesh? > > > > > > > > From my understanding, the rows handled by each processor should be consecutive, but the indices of nodes which I directly read from a gmsh file are not. Is this an issue? > > > > > > > > Best, > > > > > > > > Dongyu > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ From vaclav.hapla at erdw.ethz.ch Thu Jul 4 09:13:48 2019 From: vaclav.hapla at erdw.ethz.ch (Hapla Vaclav) Date: Thu, 4 Jul 2019 14:13:48 +0000 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: References: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> Message-ID: <569640B6-F263-4C7B-AE2E-402F196D5AB5@erdw.ethz.ch> On 4 Jul 2019, at 15:36, Matthew Knepley > wrote: On Wed, Jul 3, 2019 at 1:03 PM Fabian.Jakub > wrote: Found the change... it was in Commit 6d563f2afc7bcefff9dc7869193a6debf1d639a5 Date: Wed Jun 26 12:32:01 2019 +0000 remove remnants of non-existing DMPlexRestore{Cone,ConeOrientation,Support} You are right! Vaclav, you cannot remove the Fortran RestoreCone() because it is actually doing something. Fortran needs to create arrays, and this code deallocates them. We do not need that in C. Can you put those back? Sure, but then it should be documented somehow. Having that functions just for Fortran without explanation is very confusing. But I only removed the following interface declarations and there are no other occurrences in the whole source tree: git diff 6d563f2~ -- src/dm/f90-mod/petscdmplex.h90 diff --git a/src/dm/f90-mod/petscdmplex.h90 b/src/dm/f90-mod/petscdmplex.h90 index 7182cba..d20cdc4 100644 --- a/src/dm/f90-mod/petscdmplex.h90 +++ b/src/dm/f90-mod/petscdmplex.h90 @@ -10,16 +10,6 @@ End Interface Interface - Subroutine DMPlexRestoreCone(m,p,cone,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: cone(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetConeOrientation(m,p,coneOrient,ierr) use petscdmdef PetscInt p @@ -40,16 +30,6 @@ End Interface Interface - Subroutine DMPlexRestoreSupport(m,p,support,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: support(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetTransitiveClosure(m,p,useCone,points,ierr) use petscdmdef PetscInt p So I wonder what these can actually do? Where are the mentioned C functions? I don't see any. When we are at it, why e.g. DMPlexGetCone() has the /*@C comment on top when no custom fortran stub is provided? Vaclav Thanks, Matt The patch removed the fortran stubs but not the C functions. In this case I guess that the fortran compiler reverts to the default calling convention and thus does not nullify the cone pointers but rather nulls the values in the cone. I suggest you remove the C functions to get a compiler error or revert the path. Many thanks, Fabian On 7/3/19 3:18 PM, Matthew Knepley wrote: > On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Matt, Dear Petsc >> >> After my recent pull of petsc master I encounter test fails that use >> dmplex meshes. >> >> I attached a simple script to show an issue where DMPlexRestoreCone >> mangles the cone relationships in the DM. >> >> The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 >> @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 >> >> We recently had a system update and I am not 100% sure if this is some >> weird stuff going on at my end or if this is indeed Plex related or if I >> should not call Restore in the inner loop. >> >> Anyway, I would be most grateful if you could try to reproduce the error >> and hint me towards a solution. >> > > Hi Fabian, > > I think it might be your system update. Seems to run fine for me: > > master *:~/Downloads/tmp$ ./plex_prism > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > > Thanks, > > > Matt > > >> As always, many many thanks! >> >> Fabian >> >> P.S. here the output of >> make run >> >> and note the cones, the second time they are outputted... >> >> ./plex_prism -show_plex ::ascii_info_detail >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 1 (0) >> [0]: 0 <---- 2 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 6 (0) >> [0]: 1 <---- 7 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 12 (0) >> [0]: 2 <---- 13 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 7 (0) >> [0]: 3 <---- 9 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 6 (0) >> [0]: 4 <---- 9 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 8 (0) >> [0]: 5 <---- 10 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 15 (0) >> [0]: 6 <---- 16 (0) >> [0]: 7 <---- 15 (0) >> [0]: 7 <---- 17 (0) >> [0]: 8 <---- 16 (0) >> [0]: 8 <---- 17 (0) >> [0]: 9 <---- 15 (0) >> [0]: 9 <---- 18 (0) >> [0]: 10 <---- 16 (0) >> [0]: 10 <---- 19 (0) >> [0]: 11 <---- 17 (0) >> [0]: 11 <---- 20 (0) >> [0]: 12 <---- 18 (0) >> [0]: 12 <---- 19 (0) >> [0]: 13 <---- 18 (0) >> [0]: 13 <---- 20 (0) >> [0]: 14 <---- 19 (0) >> [0]: 14 <---- 20 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> >> >> >> >> >> >> >> >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 0 (0) >> [0]: 6 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vaclav.hapla at erdw.ethz.ch Thu Jul 4 09:19:59 2019 From: vaclav.hapla at erdw.ethz.ch (Hapla Vaclav) Date: Thu, 4 Jul 2019 14:19:59 +0000 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: <569640B6-F263-4C7B-AE2E-402F196D5AB5@erdw.ethz.ch> References: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> <569640B6-F263-4C7B-AE2E-402F196D5AB5@erdw.ethz.ch> Message-ID: <38E71180-2D4F-429B-AA6B-CD66B5D4AF94@erdw.ethz.ch> On 4 Jul 2019, at 16:13, Hapla Vaclav > wrote: On 4 Jul 2019, at 15:36, Matthew Knepley > wrote: On Wed, Jul 3, 2019 at 1:03 PM Fabian.Jakub > wrote: Found the change... it was in Commit 6d563f2afc7bcefff9dc7869193a6debf1d639a5 Date: Wed Jun 26 12:32:01 2019 +0000 remove remnants of non-existing DMPlexRestore{Cone,ConeOrientation,Support} You are right! Vaclav, you cannot remove the Fortran RestoreCone() because it is actually doing something. Fortran needs to create arrays, and this code deallocates them. We do not need that in C. Can you put those back? Sure, but then it should be documented somehow. Having that functions just for Fortran without explanation is very confusing. But I only removed the following interface declarations and there are no other occurrences in the whole source tree: git diff 6d563f2~ -- src/dm/f90-mod/petscdmplex.h90 diff --git a/src/dm/f90-mod/petscdmplex.h90 b/src/dm/f90-mod/petscdmplex.h90 index 7182cba..d20cdc4 100644 --- a/src/dm/f90-mod/petscdmplex.h90 +++ b/src/dm/f90-mod/petscdmplex.h90 @@ -10,16 +10,6 @@ End Interface Interface - Subroutine DMPlexRestoreCone(m,p,cone,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: cone(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetConeOrientation(m,p,coneOrient,ierr) use petscdmdef PetscInt p @@ -40,16 +30,6 @@ End Interface Interface - Subroutine DMPlexRestoreSupport(m,p,support,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: support(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetTransitiveClosure(m,p,useCone,points,ierr) use petscdmdef PetscInt p So I wonder what these can actually do? Where are the mentioned C functions? I don't see any. OK, I'm an idiot. I didn't realize the fortran stubs are lowercase so didn't use -i with grep :-) Still my concern about documenting these holds. There should at least be a Fortran Note in that manpage. Thanks, Vaclav When we are at it, why e.g. DMPlexGetCone() has the /*@C comment on top when no custom fortran stub is provided? Vaclav Thanks, Matt The patch removed the fortran stubs but not the C functions. In this case I guess that the fortran compiler reverts to the default calling convention and thus does not nullify the cone pointers but rather nulls the values in the cone. I suggest you remove the C functions to get a compiler error or revert the path. Many thanks, Fabian On 7/3/19 3:18 PM, Matthew Knepley wrote: > On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Matt, Dear Petsc >> >> After my recent pull of petsc master I encounter test fails that use >> dmplex meshes. >> >> I attached a simple script to show an issue where DMPlexRestoreCone >> mangles the cone relationships in the DM. >> >> The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 >> @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 >> >> We recently had a system update and I am not 100% sure if this is some >> weird stuff going on at my end or if this is indeed Plex related or if I >> should not call Restore in the inner loop. >> >> Anyway, I would be most grateful if you could try to reproduce the error >> and hint me towards a solution. >> > > Hi Fabian, > > I think it might be your system update. Seems to run fine for me: > > master *:~/Downloads/tmp$ ./plex_prism > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > > Thanks, > > > Matt > > >> As always, many many thanks! >> >> Fabian >> >> P.S. here the output of >> make run >> >> and note the cones, the second time they are outputted... >> >> ./plex_prism -show_plex ::ascii_info_detail >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 1 (0) >> [0]: 0 <---- 2 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 6 (0) >> [0]: 1 <---- 7 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 12 (0) >> [0]: 2 <---- 13 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 7 (0) >> [0]: 3 <---- 9 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 6 (0) >> [0]: 4 <---- 9 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 8 (0) >> [0]: 5 <---- 10 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 15 (0) >> [0]: 6 <---- 16 (0) >> [0]: 7 <---- 15 (0) >> [0]: 7 <---- 17 (0) >> [0]: 8 <---- 16 (0) >> [0]: 8 <---- 17 (0) >> [0]: 9 <---- 15 (0) >> [0]: 9 <---- 18 (0) >> [0]: 10 <---- 16 (0) >> [0]: 10 <---- 19 (0) >> [0]: 11 <---- 17 (0) >> [0]: 11 <---- 20 (0) >> [0]: 12 <---- 18 (0) >> [0]: 12 <---- 19 (0) >> [0]: 13 <---- 18 (0) >> [0]: 13 <---- 20 (0) >> [0]: 14 <---- 19 (0) >> [0]: 14 <---- 20 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> >> >> >> >> >> >> >> >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 0 (0) >> [0]: 6 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vaclav.hapla at erdw.ethz.ch Thu Jul 4 09:46:33 2019 From: vaclav.hapla at erdw.ethz.ch (Hapla Vaclav) Date: Thu, 4 Jul 2019 14:46:33 +0000 Subject: [petsc-users] Issue with DMPlexRestoreCone from Fortran In-Reply-To: <38E71180-2D4F-429B-AA6B-CD66B5D4AF94@erdw.ethz.ch> References: <5d0cb6ac-c933-bab8-112a-de06f998ac0a@physik.uni-muenchen.de> <569640B6-F263-4C7B-AE2E-402F196D5AB5@erdw.ethz.ch> <38E71180-2D4F-429B-AA6B-CD66B5D4AF94@erdw.ethz.ch> Message-ID: On 4 Jul 2019, at 16:19, Hapla Vaclav > wrote: On 4 Jul 2019, at 16:13, Hapla Vaclav > wrote: On 4 Jul 2019, at 15:36, Matthew Knepley > wrote: On Wed, Jul 3, 2019 at 1:03 PM Fabian.Jakub > wrote: Found the change... it was in Commit 6d563f2afc7bcefff9dc7869193a6debf1d639a5 Date: Wed Jun 26 12:32:01 2019 +0000 remove remnants of non-existing DMPlexRestore{Cone,ConeOrientation,Support} You are right! Vaclav, you cannot remove the Fortran RestoreCone() because it is actually doing something. Fortran needs to create arrays, and this code deallocates them. We do not need that in C. Can you put those back? Sure, but then it should be documented somehow. Having that functions just for Fortran without explanation is very confusing. But I only removed the following interface declarations and there are no other occurrences in the whole source tree: git diff 6d563f2~ -- src/dm/f90-mod/petscdmplex.h90 diff --git a/src/dm/f90-mod/petscdmplex.h90 b/src/dm/f90-mod/petscdmplex.h90 index 7182cba..d20cdc4 100644 --- a/src/dm/f90-mod/petscdmplex.h90 +++ b/src/dm/f90-mod/petscdmplex.h90 @@ -10,16 +10,6 @@ End Interface Interface - Subroutine DMPlexRestoreCone(m,p,cone,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: cone(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetConeOrientation(m,p,coneOrient,ierr) use petscdmdef PetscInt p @@ -40,16 +30,6 @@ End Interface Interface - Subroutine DMPlexRestoreSupport(m,p,support,ierr) - use petscdmdef - PetscInt p - PetscInt, pointer :: support(:) - PetscErrorCode ierr - DM m - End Subroutine - End Interface - - Interface Subroutine DMPlexGetTransitiveClosure(m,p,useCone,points,ierr) use petscdmdef PetscInt p So I wonder what these can actually do? Where are the mentioned C functions? I don't see any. OK, I'm an idiot. I didn't realize the fortran stubs are lowercase so didn't use -i with grep :-) Still my concern about documenting these holds. There should at least be a Fortran Note in that manpage. Please discuss further at https://bitbucket.org/petsc/petsc/pull-requests/1847 Thanks, Vaclav When we are at it, why e.g. DMPlexGetCone() has the /*@C comment on top when no custom fortran stub is provided? Vaclav Thanks, Matt The patch removed the fortran stubs but not the C functions. In this case I guess that the fortran compiler reverts to the default calling convention and thus does not nullify the cone pointers but rather nulls the values in the cone. I suggest you remove the C functions to get a compiler error or revert the path. Many thanks, Fabian On 7/3/19 3:18 PM, Matthew Knepley wrote: > On Wed, Jul 3, 2019 at 7:38 AM Fabian.Jakub via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Matt, Dear Petsc >> >> After my recent pull of petsc master I encounter test fails that use >> dmplex meshes. >> >> I attached a simple script to show an issue where DMPlexRestoreCone >> mangles the cone relationships in the DM. >> >> The behaviour happens with gcc 7.4.0 aswell as icc 19.0.3.199 >> @ current petsc/master d3dc048301fdb3e4c282b528e7b28910551cfe13 >> >> We recently had a system update and I am not 100% sure if this is some >> weird stuff going on at my end or if this is indeed Plex related or if I >> should not call Restore in the inner loop. >> >> Anyway, I would be most grateful if you could try to reproduce the error >> and hint me towards a solution. >> > > Hi Fabian, > > I think it might be your system update. Seems to run fine for me: > > master *:~/Downloads/tmp$ ./plex_prism > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > master *:~/Downloads/tmp$ ./plex_prism -show_plex ::ascii_info_detail > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > Cone 0 : 1 2 3 4 > 5 > > Cone 1 : 6 7 8 > > Cone 2 : 12 13 14 > > Cone 3 : 7 9 11 13 > > Cone 4 : 6 9 10 12 > > Cone 5 : 8 10 11 14 > > Cone 6 : 15 16 > > Cone 7 : 15 17 > > Cone 8 : 16 17 > > Cone 9 : 15 18 > > Cone 10 : 16 19 > > Cone 11 : 17 20 > > Cone 12 : 18 19 > > Cone 13 : 18 20 > > Cone 14 : 19 20 > > Cone 15 : > > Cone 16 : > > Cone 17 : > > Cone 18 : > > Cone 19 : > > Cone 20 : > > DM Object: testplex 1 MPI processes > > type: plex > > testplex in 3 dimensions: > > Supports: > > [0] Max support size: 3 > > [0]: 1 ----> 0 > > [0]: 2 ----> 0 > > [0]: 3 ----> 0 > > [0]: 4 ----> 0 > > [0]: 5 ----> 0 > > [0]: 6 ----> 1 > > [0]: 6 ----> 4 > > [0]: 7 ----> 1 > > [0]: 7 ----> 3 > > [0]: 8 ----> 1 > > [0]: 8 ----> 5 > > [0]: 9 ----> 3 > > [0]: 9 ----> 4 > > [0]: 10 ----> 4 > > [0]: 10 ----> 5 > > [0]: 11 ----> 3 > > [0]: 11 ----> 5 > > [0]: 12 ----> 2 > > [0]: 12 ----> 4 > > [0]: 13 ----> 2 > > [0]: 13 ----> 3 > > [0]: 14 ----> 2 > > [0]: 14 ----> 5 > > [0]: 15 ----> 6 > > [0]: 15 ----> 7 > > [0]: 15 ----> 9 > > [0]: 16 ----> 6 > > [0]: 16 ----> 8 > > [0]: 16 ----> 10 > > [0]: 17 ----> 7 > > [0]: 17 ----> 8 > > [0]: 17 ----> 11 > > [0]: 18 ----> 9 > > [0]: 18 ----> 12 > > [0]: 18 ----> 13 > > [0]: 19 ----> 10 > > [0]: 19 ----> 12 > > [0]: 19 ----> 14 > > [0]: 20 ----> 11 > > [0]: 20 ----> 13 > > [0]: 20 ----> 14 > > Cones: > > [0] Max cone size: 5 > > [0]: 0 <---- 1 (0) > > [0]: 0 <---- 2 (0) > > [0]: 0 <---- 3 (0) > > [0]: 0 <---- 4 (0) > > [0]: 0 <---- 5 (0) > > [0]: 1 <---- 6 (0) > > [0]: 1 <---- 7 (0) > > [0]: 1 <---- 8 (0) > > [0]: 2 <---- 12 (0) > > [0]: 2 <---- 13 (0) > > [0]: 2 <---- 14 (0) > > [0]: 3 <---- 7 (0) > > [0]: 3 <---- 9 (0) > > [0]: 3 <---- 11 (0) > > [0]: 3 <---- 13 (0) > > [0]: 4 <---- 6 (0) > > [0]: 4 <---- 9 (0) > > [0]: 4 <---- 10 (0) > > [0]: 4 <---- 12 (0) > > [0]: 5 <---- 8 (0) > > [0]: 5 <---- 10 (0) > > [0]: 5 <---- 11 (0) > > [0]: 5 <---- 14 (0) > > [0]: 6 <---- 15 (0) > > [0]: 6 <---- 16 (0) > > [0]: 7 <---- 15 (0) > > [0]: 7 <---- 17 (0) > > [0]: 8 <---- 16 (0) > > [0]: 8 <---- 17 (0) > > [0]: 9 <---- 15 (0) > > [0]: 9 <---- 18 (0) > > [0]: 10 <---- 16 (0) > > [0]: 10 <---- 19 (0) > > [0]: 11 <---- 17 (0) > > [0]: 11 <---- 20 (0) > > [0]: 12 <---- 18 (0) > > [0]: 12 <---- 19 (0) > > [0]: 13 <---- 18 (0) > > [0]: 13 <---- 20 (0) > > [0]: 14 <---- 19 (0) > > [0]: 14 <---- 20 (0) > > coordinates with 1 fields > > field 0 with 3 components > > Process 0: > > ( 0) dim 0 offset 0 > > ( 1) dim 0 offset 0 > > ( 2) dim 0 offset 0 > > ( 3) dim 0 offset 0 > > ( 4) dim 0 offset 0 > > ( 5) dim 0 offset 0 > > ( 6) dim 0 offset 0 > > ( 7) dim 0 offset 0 > > ( 8) dim 0 offset 0 > > ( 9) dim 0 offset 0 > > ( 10) dim 0 offset 0 > > ( 11) dim 0 offset 0 > > ( 12) dim 0 offset 0 > > ( 13) dim 0 offset 0 > > ( 14) dim 0 offset 0 > > ( 15) dim 3 offset 0 0. 0. 0. > > ( 16) dim 3 offset 3 0. 2. 0. > > ( 17) dim 3 offset 6 2. 1. 0. > > ( 18) dim 3 offset 9 0. 0. 1. > > ( 19) dim 3 offset 12 0. 2. 1. > > ( 20) dim 3 offset 15 2. 1. 1. > > > Thanks, > > > Matt > > >> As always, many many thanks! >> >> Fabian >> >> P.S. here the output of >> make run >> >> and note the cones, the second time they are outputted... >> >> ./plex_prism -show_plex ::ascii_info_detail >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 1 (0) >> [0]: 0 <---- 2 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 6 (0) >> [0]: 1 <---- 7 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 12 (0) >> [0]: 2 <---- 13 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 7 (0) >> [0]: 3 <---- 9 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 6 (0) >> [0]: 4 <---- 9 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 8 (0) >> [0]: 5 <---- 10 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 15 (0) >> [0]: 6 <---- 16 (0) >> [0]: 7 <---- 15 (0) >> [0]: 7 <---- 17 (0) >> [0]: 8 <---- 16 (0) >> [0]: 8 <---- 17 (0) >> [0]: 9 <---- 15 (0) >> [0]: 9 <---- 18 (0) >> [0]: 10 <---- 16 (0) >> [0]: 10 <---- 19 (0) >> [0]: 11 <---- 17 (0) >> [0]: 11 <---- 20 (0) >> [0]: 12 <---- 18 (0) >> [0]: 12 <---- 19 (0) >> [0]: 13 <---- 18 (0) >> [0]: 13 <---- 20 (0) >> [0]: 14 <---- 19 (0) >> [0]: 14 <---- 20 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> >> >> >> >> >> >> >> >> DM Object: testplex 1 MPI processes >> type: plex >> testplex in 3 dimensions: >> Supports: >> [0] Max support size: 3 >> [0]: 1 ----> 0 >> [0]: 2 ----> 0 >> [0]: 3 ----> 0 >> [0]: 4 ----> 0 >> [0]: 5 ----> 0 >> [0]: 6 ----> 1 >> [0]: 6 ----> 4 >> [0]: 7 ----> 1 >> [0]: 7 ----> 3 >> [0]: 8 ----> 1 >> [0]: 8 ----> 5 >> [0]: 9 ----> 3 >> [0]: 9 ----> 4 >> [0]: 10 ----> 4 >> [0]: 10 ----> 5 >> [0]: 11 ----> 3 >> [0]: 11 ----> 5 >> [0]: 12 ----> 2 >> [0]: 12 ----> 4 >> [0]: 13 ----> 2 >> [0]: 13 ----> 3 >> [0]: 14 ----> 2 >> [0]: 14 ----> 5 >> [0]: 15 ----> 6 >> [0]: 15 ----> 7 >> [0]: 15 ----> 9 >> [0]: 16 ----> 6 >> [0]: 16 ----> 8 >> [0]: 16 ----> 10 >> [0]: 17 ----> 7 >> [0]: 17 ----> 8 >> [0]: 17 ----> 11 >> [0]: 18 ----> 9 >> [0]: 18 ----> 12 >> [0]: 18 ----> 13 >> [0]: 19 ----> 10 >> [0]: 19 ----> 12 >> [0]: 19 ----> 14 >> [0]: 20 ----> 11 >> [0]: 20 ----> 13 >> [0]: 20 ----> 14 >> Cones: >> [0] Max cone size: 5 >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 0 (0) >> [0]: 0 <---- 3 (0) >> [0]: 0 <---- 4 (0) >> [0]: 0 <---- 5 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 0 (0) >> [0]: 1 <---- 8 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 0 (0) >> [0]: 2 <---- 14 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 0 (0) >> [0]: 3 <---- 11 (0) >> [0]: 3 <---- 13 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 0 (0) >> [0]: 4 <---- 10 (0) >> [0]: 4 <---- 12 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 0 (0) >> [0]: 5 <---- 11 (0) >> [0]: 5 <---- 14 (0) >> [0]: 6 <---- 0 (0) >> [0]: 6 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 7 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 8 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 9 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 10 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 11 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 12 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 13 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> [0]: 14 <---- 0 (0) >> coordinates with 1 fields >> field 0 with 3 components >> Process 0: >> ( 0) dim 0 offset 0 >> ( 1) dim 0 offset 0 >> ( 2) dim 0 offset 0 >> ( 3) dim 0 offset 0 >> ( 4) dim 0 offset 0 >> ( 5) dim 0 offset 0 >> ( 6) dim 0 offset 0 >> ( 7) dim 0 offset 0 >> ( 8) dim 0 offset 0 >> ( 9) dim 0 offset 0 >> ( 10) dim 0 offset 0 >> ( 11) dim 0 offset 0 >> ( 12) dim 0 offset 0 >> ( 13) dim 0 offset 0 >> ( 14) dim 0 offset 0 >> ( 15) dim 3 offset 0 0. 0. 0. >> ( 16) dim 3 offset 3 0. 2. 0. >> ( 17) dim 3 offset 6 2. 1. 0. >> ( 18) dim 3 offset 9 0. 0. 1. >> ( 19) dim 3 offset 12 0. 2. 1. >> ( 20) dim 3 offset 15 2. 1. 1. >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From davelee2804 at gmail.com Thu Jul 4 19:29:29 2019 From: davelee2804 at gmail.com (Dave Lee) Date: Fri, 5 Jul 2019 10:29:29 +1000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() Message-ID: Hi PETSc, I have a problem for which I need to exclude certain degrees of freedom from the evaluation of the norm as used to monitor the residual in FGMRES (don't ask!). Therefore I would like to replace the existing norm evaluation with my own version. Currently this is done within: KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) as: *res = PetscAbsScalar(*RS(it+1)); Firstly, I would like to make sure I can replicate the existing residual norm. I tried to do this as: KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); VecNorm(tmp_sol,NORM_2,&res_tmp); VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); res_norm = fabs(res_tmp-res_rhs); But this doesn't match the norm that comes out of UpdateHessenberg. Any ideas on how I can re-create the norm derived from UpdateHessenberg as an expansion over the Kyrlov vectors in FGMRES? Cheers, Dave. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 4 19:55:32 2019 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jul 2019 19:55:32 -0500 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: References: Message-ID: On Thu, Jul 4, 2019 at 7:30 PM Dave Lee via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi PETSc, > > I have a problem for which I need to exclude certain degrees of freedom > from the evaluation of the norm as used to monitor the residual in FGMRES > (don't ask!). Therefore I would like to replace the existing norm > evaluation with my own version. > > Currently this is done within: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > as: > *res = PetscAbsScalar(*RS(it+1)); > > Firstly, I would like to make sure I can replicate the existing residual > norm. I tried to do this as: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); > VecNorm(tmp_sol,NORM_2,&res_tmp); > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > res_norm = fabs(res_tmp-res_rhs); > > But this doesn't match the norm that comes out of UpdateHessenberg. > > Any ideas on how I can re-create the norm derived from UpdateHessenberg as > an expansion over the Kyrlov vectors in FGMRES? > Is there a reason you cannot just calculate the norm you want in a custom convergence criterion? https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetConvergenceTest.html Thanks, Matt > Cheers, Dave. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 4 21:59:38 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 5 Jul 2019 02:59:38 +0000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: References: Message-ID: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users wrote: > > Hi PETSc, > > I have a problem for which I need to exclude certain degrees of freedom from the evaluation of the norm as used to monitor the residual in FGMRES (don't ask!). Therefore I would like to replace the existing norm evaluation with my own version. > > Currently this is done within: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > as: > *res = PetscAbsScalar(*RS(it+1)); > > Firstly, I would like to make sure I can replicate the existing residual norm. I tried to do this as: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); I don't understand the lines of code below. I don't see how they could compute the norm of the residual. >From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned residual with ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); ierr = VecDestroy(&resid);CHKERRQ(ierr); since FGMRES always uses right preconditioning (and hence the unpreconditioned residual) this should match the value computed internally by FGMRES (subject to roundoff differences) To exclude certain values in the residual from the norm calculation you could write a custom "VecNorm" norm that skipped those entries. Barry > VecNorm(tmp_sol,NORM_2,&res_tmp); > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > res_norm = fabs(res_tmp-res_rhs); > > But this doesn't match the norm that comes out of UpdateHessenberg. > > Any ideas on how I can re-create the norm derived from UpdateHessenberg as an expansion over the Kyrlov vectors in FGMRES? > > Cheers, Dave. From hongzhang at anl.gov Thu Jul 4 23:18:20 2019 From: hongzhang at anl.gov (Zhang, Hong) Date: Fri, 5 Jul 2019 04:18:20 +0000 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: Message-ID: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> On Jul 3, 2019, at 3:10 PM, Manuel Valera > wrote: Thanks Zhang for your answer, I ended up getting a compiling and running TS routine... that does not give me the answers, and i am not sure what i am doing wrong, My TS code so far looks like this: (...initialization...) call TSCreate(PETSC_COMM_WORLD,ts,ierr) call TSSetProblemType(ts,TS_NONLINEAR,ierr) call TSSetSolution(ts,gTemperature,ierr) call TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) The second last argument should be the user context. If you pass NULL, the ctx variable in your FormRHSFunction will be NULL. call TSSetType(ts,TSSSP,ierr) call TSSetTimeStep(ts,dt,ierr) call TSSetDM(ts,daScalars,ierr) call TSSetMaxTime(ts,TotalTime,ierr) call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) call TSSetFromOptions(ts,ierr) call TSSetUp(ts,ierr) (... then inside main calculations loop i have...) call TSSolve(ts,gTemperature,ierr) (...and my RHSFunction looks like this...) subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) use petscvec use petscts use petscdmda use petscdm USE utils USE dmdaobjs USE probsize USE modelparams implicit none TS :: ts PetscReal :: t Vec :: ctx,t_loc,rhs_loc PetscErrorCode :: ierr call TSGetDM(ts,daScalars,ierr) call DMGetLocalVector(daScalars,t_loc,ierr); t_loc is the input, it should be not modified. call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you want to scatter ghost points for t_loc, not gTemperature. call DMDASolveTExplicit(3) call DMGetLocalVector(daScalars,rhs_loc,ierr); call DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) call DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) There is no need to scatter ghost points for rhs_loc. call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) end subroutine FormRHSFunction Where DMDASolveTExplicit(3) is the old function to calculate time integration with runge kutta, modified to only generate the f(t,u) which in this case is rhs_loc, I still have several doubts: * Will this explicit RHS calculation work with TS routines? my time integration is explicit so far and it would involve a fair deal of extra work to make it implicit. For explicit time integration, one needs to provide only RHSFunction. * This 't' parameter in the RHSFunction is controlled by PETSC? i am not providing anything for it directly, where is it coming from? It is controlled by PETSc. If your problem is autonomous (the RHS does not depend on t), it can be simply ignored. * Do i need to provide a Jacobian or the TS routine will try to guess one? This is related to the first point where my time scheme being explicit does not use a jacobian. For explicit time integration, Jacobian is not needed. Hong Thanks, any help is appreciated, if you see any errors or need more information please let me know, Regards, Manuel On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong > wrote: On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users > wrote: Hi PETSc, I am trying to implement the Time stepping routines in my model, i have a working runge-kutta time scheme that goes to the following steps: * Interpolate u,v,w to the time advancing variable position. * Calculate nonlinear coefficients and advect velocities with a forward-backward shock capturing scheme. * Calculate the variable laplacian * Sum terms to create RHS (nonlinear advected velocities + laplacian) * Finally, the runge kutta integration is done in a typical way that looks like: newtemp(t+1) = prevtemp(t) + dt*RHS So my questions are: * I think my problem is nonlinear, but is being made linearized by the advecting scheme, is this correct? this is to know if i should use the linear or nonlinear routines for TS. TSComputeRHSFunctionLinear is just a convenience function for linear ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, it is fine to assume your problem is nonlinear and think of the form udot=f(t,u) where f is the RHS function. * How do i know what are the appropriate routines i should be using here? so far i think i should use the following: call TSCreate(PETSC_COMM_WORLD,ts,ierr) call TSSetProblemType(ts,TS_LINEAR,ierr) call TSSetTimeStep(ts,dt,ierr) call TSSetFromOptions(ts,ierr) call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) call TSSolve(ts,loctemperature,ierr) Should i use call TSSetRHSJacobian for the temperature jacobian in this case? I would suggest to write your own RHSFunction and set it to TS with TSSetRHSFunction(). I am using https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html as a general guide, is there a more appropriate example? ex4 is a good example. In addition, ex9 uses finite volume method with slope limiters to solve a variety of problems such as advection equation, burgers equation and shallow water equation. It might be an overkill, but it seems to be close to the problem you are trying to solve. Note that you might want to use the SSP methods (-ts_type ssp) instead of the classic Runge-Kutta methods for problems with shocks. Hong (Mr.) The dt value and total timesteps are controlled by the model, Thanks for your help, -------------- next part -------------- An HTML attachment was scrubbed... URL: From davelee2804 at gmail.com Thu Jul 4 23:22:39 2019 From: davelee2804 at gmail.com (Dave Lee) Date: Fri, 5 Jul 2019 14:22:39 +1000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> References: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> Message-ID: Hi Matt and Barry, thanks for the good ideas. I wasn't aware of the function KSPSetConvergenceTest(), thanks for alerting me to this Matt. Unfortunately I really do need to exclude a couple of specific degrees of freedom from the norm, and as such I want to make sure I can replicate the norm as computed within UpdateHessenberg() before I replace this. Apologies Barry, there was a stupid bug in that code, it should have read: KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); KSPFGMRESBuildSoln(fgmres->nrs,ksp->vec_sol,tmp_sol,ksp,loc_it+1); VecAXPY(tmp_sol,-1.0,ksp->vec_rhs); VecNorm(tmp_sol,NORM_2,&res_tmp); Which does not give the same result as the norm for UpdateHessenberg(). Unfortunately I want to avoid using KSPBuildResidual(ksp,NULL,NULL,&resid); as this calls KSP_MatMult(), which will in turn call my snes residual assembly routine. This routine in turn calls a Navier Stokes solver to run for a fixed (long) time period, and so is expensive. Replacing VecNorm() with my own version is indeed what I'm doing elsewhere, I'm just not sure of how to replace the UpdateHessenberg() norm right now. Cheers, Dave. On Fri, Jul 5, 2019 at 12:59 PM Smith, Barry F. wrote: > > > > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hi PETSc, > > > > I have a problem for which I need to exclude certain degrees of freedom > from the evaluation of the norm as used to monitor the residual in FGMRES > (don't ask!). Therefore I would like to replace the existing norm > evaluation with my own version. > > > > Currently this is done within: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > > as: > > *res = PetscAbsScalar(*RS(it+1)); > > > > Firstly, I would like to make sure I can replicate the existing residual > norm. I tried to do this as: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); > > I don't understand the lines of code below. I don't see how they could > compute the norm of the residual. > > From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned > residual with > > ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); > ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); > ierr = VecDestroy(&resid);CHKERRQ(ierr); > > since FGMRES always uses right preconditioning (and hence the > unpreconditioned residual) this should match the value computed internally > by FGMRES (subject to roundoff differences) > > To exclude certain values in the residual from the norm calculation you > could write a custom "VecNorm" norm that skipped those entries. > > Barry > > > > > > > VecNorm(tmp_sol,NORM_2,&res_tmp); > > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > > res_norm = fabs(res_tmp-res_rhs); > > > > But this doesn't match the norm that comes out of UpdateHessenberg. > > > > Any ideas on how I can re-create the norm derived from UpdateHessenberg > as an expansion over the Kyrlov vectors in FGMRES? > > > > Cheers, Dave. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jul 5 00:24:43 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 5 Jul 2019 05:24:43 +0000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: References: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> Message-ID: <1AFFE94A-E200-4161-8694-BD6E4D19C3CA@mcs.anl.gov> > On Jul 4, 2019, at 11:22 PM, Dave Lee wrote: > > Hi Matt and Barry, > > thanks for the good ideas. > > I wasn't aware of the function KSPSetConvergenceTest(), thanks for alerting me to this Matt. Unfortunately I really do need to exclude a couple of specific degrees of freedom from the norm, and as such I want to make sure I can replicate the norm as computed within UpdateHessenberg() before I replace this. > > Apologies Barry, there was a stupid bug in that code, it should have read: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > KSPFGMRESBuildSoln(fgmres->nrs,ksp->vec_sol,tmp_sol,ksp,loc_it+1); > VecAXPY(tmp_sol,-1.0,ksp->vec_rhs); > VecNorm(tmp_sol,NORM_2,&res_tmp); This won't work. > > Which does not give the same result as the norm for UpdateHessenberg(). Unfortunately I want to avoid using > KSPBuildResidual(ksp,NULL,NULL,&resid); > as this calls KSP_MatMult(), which will in turn call my snes residual assembly routine. This routine in turn calls a Navier Stokes solver to run for a fixed (long) time period, and so is expensive. If you want a "freshly" computed residual you have no choice but to call MatMult() > > Replacing VecNorm() with my own version is indeed what I'm doing elsewhere, I'm just not sure of how to replace the UpdateHessenberg() norm right now. It sounds like you don't want FGMRES to "see" some of your equations at all? You could do this by taking the result of your true matrix-vector product and zeroing the values for the equation you don't want FGMRES to "see". This will introduce a null space for your linear operator but that may not matter. Of course since FGMRES doesn't "see" these components it will make no effort to make them small so the residual for those equations will be whatever which means that the solution to your linear system is not the solution to the original system. Barry > > Cheers, Dave. > > On Fri, Jul 5, 2019 at 12:59 PM Smith, Barry F. wrote: > > > > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users wrote: > > > > Hi PETSc, > > > > I have a problem for which I need to exclude certain degrees of freedom from the evaluation of the norm as used to monitor the residual in FGMRES (don't ask!). Therefore I would like to replace the existing norm evaluation with my own version. > > > > Currently this is done within: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > > as: > > *res = PetscAbsScalar(*RS(it+1)); > > > > Firstly, I would like to make sure I can replicate the existing residual norm. I tried to do this as: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); > > I don't understand the lines of code below. I don't see how they could compute the norm of the residual. > > From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned residual with > > ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); > ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); > ierr = VecDestroy(&resid);CHKERRQ(ierr); > > since FGMRES always uses right preconditioning (and hence the unpreconditioned residual) this should match the value computed internally by FGMRES (subject to roundoff differences) > > To exclude certain values in the residual from the norm calculation you could write a custom "VecNorm" norm that skipped those entries. > > Barry > > > > > > > VecNorm(tmp_sol,NORM_2,&res_tmp); > > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > > res_norm = fabs(res_tmp-res_rhs); > > > > But this doesn't match the norm that comes out of UpdateHessenberg. > > > > Any ideas on how I can re-create the norm derived from UpdateHessenberg as an expansion over the Kyrlov vectors in FGMRES? > > > > Cheers, Dave. > From jcrean01 at gmail.com Thu Jul 4 16:05:32 2019 From: jcrean01 at gmail.com (Jared Crean) Date: Thu, 4 Jul 2019 17:05:32 -0400 Subject: [petsc-users] LU Performance Message-ID: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> Hello, ??? I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. ??? I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc.? The matrix data files are not included because they are about 250 megabytes.? I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu).? Both results show nearly all of the time is spent in MatLuFactorNum.? The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. ??? Jared Crean -------------- next part -------------- A non-text attachment was scrubbed... Name: test_files.tar.gz Type: application/gzip Size: 5425 bytes Desc: not available URL: From dave.mayhem23 at gmail.com Fri Jul 5 01:33:25 2019 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 5 Jul 2019 08:33:25 +0200 Subject: [petsc-users] LU Performance In-Reply-To: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> Message-ID: On Fri, 5 Jul 2019 at 08:19, Jared Crean via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I am getting very bad performance from the Umfpack LU solver when I > use it via Petsc compared to calling Umfpack directly. It takes about > 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds > when I use Petsc with -ksp_type preonly -pc_type lu > -pc_factor_mat_solver_type umfpack. > > I have attached a minimal example (test.c) that reads a matrix from > a file, solves with Umfpack, and then solves with Petsc. The matrix > data files are not included because they are about 250 megabytes. I > also attached the output of the program with -log_view for > -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and > -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show > nearly all of the time is spent in MatLuFactorNum. The times are very > similar, so I am wondering if Petsc is really calling Umfpack or if the > Petsc LU solver is getting called in both cases. Are you using the same umfpack build for both tests (stand alone umfpack and petsc+umfpack)? If no, are you certain that the umfpack petsc built links against the same threaded blas library as the umfpack build which gives the fast result? > > > Jared Crean > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jul 5 03:02:23 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 5 Jul 2019 08:02:23 +0000 Subject: [petsc-users] LU Performance In-Reply-To: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> Message-ID: When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. Barry > On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users wrote: > > Hello, > > I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. > > I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. > > > Jared Crean > > From davelee2804 at gmail.com Fri Jul 5 06:29:07 2019 From: davelee2804 at gmail.com (Dave Lee) Date: Fri, 5 Jul 2019 21:29:07 +1000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: <1AFFE94A-E200-4161-8694-BD6E4D19C3CA@mcs.anl.gov> References: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> <1AFFE94A-E200-4161-8694-BD6E4D19C3CA@mcs.anl.gov> Message-ID: Thanks Barry, good to know that MatMult() is necessary to get a fresh residual estimation. The only thing I don't want to "see" all of the equations is the residual computation. Replacing VecNorm() with a custom version seems to be fine everywhere else, but the UpdateHessenberg() call is problematic. I will have a think about Matt's idea of customising the convergence test. Thanks again, Dave. On Fri, Jul 5, 2019 at 3:24 PM Smith, Barry F. wrote: > > > > On Jul 4, 2019, at 11:22 PM, Dave Lee wrote: > > > > Hi Matt and Barry, > > > > thanks for the good ideas. > > > > I wasn't aware of the function KSPSetConvergenceTest(), thanks for > alerting me to this Matt. Unfortunately I really do need to exclude a > couple of specific degrees of freedom from the norm, and as such I want to > make sure I can replicate the norm as computed within UpdateHessenberg() > before I replace this. > > > > Apologies Barry, there was a stupid bug in that code, it should have > read: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > > KSPFGMRESBuildSoln(fgmres->nrs,ksp->vec_sol,tmp_sol,ksp,loc_it+1); > > VecAXPY(tmp_sol,-1.0,ksp->vec_rhs); > > VecNorm(tmp_sol,NORM_2,&res_tmp); > > This won't work. > > > > Which does not give the same result as the norm for UpdateHessenberg(). > Unfortunately I want to avoid using > > KSPBuildResidual(ksp,NULL,NULL,&resid); > > as this calls KSP_MatMult(), which will in turn call my snes residual > assembly routine. This routine in turn calls a Navier Stokes solver to run > for a fixed (long) time period, and so is expensive. > > If you want a "freshly" computed residual you have no choice but to > call MatMult() > > > > Replacing VecNorm() with my own version is indeed what I'm doing > elsewhere, I'm just not sure of how to replace the UpdateHessenberg() norm > right now. > > It sounds like you don't want FGMRES to "see" some of your equations at > all? You could do this by taking the result of your true matrix-vector > product and zeroing the values for the equation you don't want FGMRES to > "see". This will introduce a null space for your linear operator but that > may not matter. Of course since FGMRES doesn't "see" these components it > will make no effort to make them small so the residual for those equations > will be whatever which means that the solution to your linear system is not > the solution to the original system. > > Barry > > > > > > Cheers, Dave. > > > > On Fri, Jul 5, 2019 at 12:59 PM Smith, Barry F. > wrote: > > > > > > > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > > > Hi PETSc, > > > > > > I have a problem for which I need to exclude certain degrees of > freedom from the evaluation of the norm as used to monitor the residual in > FGMRES (don't ask!). Therefore I would like to replace the existing norm > evaluation with my own version. > > > > > > Currently this is done within: > > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > > > as: > > > *res = PetscAbsScalar(*RS(it+1)); > > > > > > Firstly, I would like to make sure I can replicate the existing > residual norm. I tried to do this as: > > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > > > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); > > > > I don't understand the lines of code below. I don't see how they > could compute the norm of the residual. > > > > From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned > residual with > > > > ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); > > ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); > > ierr = VecDestroy(&resid);CHKERRQ(ierr); > > > > since FGMRES always uses right preconditioning (and hence the > unpreconditioned residual) this should match the value computed internally > by FGMRES (subject to roundoff differences) > > > > To exclude certain values in the residual from the norm calculation you > could write a custom "VecNorm" norm that skipped those entries. > > > > Barry > > > > > > > > > > > > > VecNorm(tmp_sol,NORM_2,&res_tmp); > > > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > > > res_norm = fabs(res_tmp-res_rhs); > > > > > > But this doesn't match the norm that comes out of UpdateHessenberg. > > > > > > Any ideas on how I can re-create the norm derived from > UpdateHessenberg as an expansion over the Kyrlov vectors in FGMRES? > > > > > > Cheers, Dave. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 5 08:02:45 2019 From: jed at jedbrown.org (Jed Brown) Date: Fri, 05 Jul 2019 07:02:45 -0600 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: References: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> Message-ID: <87bly8ws5m.fsf@jedbrown.org> Dave, have you considered using GCR instead of FGMRES? It's a flexible method that is equivalent in many circumstances, but provides the residual and solution at each iteration without needing to "build" it. Dave Lee via petsc-users writes: > Hi Matt and Barry, > > thanks for the good ideas. > > I wasn't aware of the function KSPSetConvergenceTest(), thanks for alerting > me to this Matt. Unfortunately I really do need to exclude a couple of > specific degrees of freedom from the norm, and as such I want to make sure > I can replicate the norm as computed within UpdateHessenberg() before I > replace this. > > Apologies Barry, there was a stupid bug in that code, it should have read: > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > KSPFGMRESBuildSoln(fgmres->nrs,ksp->vec_sol,tmp_sol,ksp,loc_it+1); > VecAXPY(tmp_sol,-1.0,ksp->vec_rhs); > VecNorm(tmp_sol,NORM_2,&res_tmp); > > Which does not give the same result as the norm for UpdateHessenberg(). > Unfortunately I want to avoid using > KSPBuildResidual(ksp,NULL,NULL,&resid); > as this calls KSP_MatMult(), which will in turn call my snes residual > assembly routine. This routine in turn calls a Navier Stokes solver to run > for a fixed (long) time period, and so is expensive. > > Replacing VecNorm() with my own version is indeed what I'm doing elsewhere, > I'm just not sure of how to replace the UpdateHessenberg() norm right now. > > Cheers, Dave. > > On Fri, Jul 5, 2019 at 12:59 PM Smith, Barry F. wrote: > >> >> >> > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > >> > Hi PETSc, >> > >> > I have a problem for which I need to exclude certain degrees of freedom >> from the evaluation of the norm as used to monitor the residual in FGMRES >> (don't ask!). Therefore I would like to replace the existing norm >> evaluation with my own version. >> > >> > Currently this is done within: >> > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) >> > as: >> > *res = PetscAbsScalar(*RS(it+1)); >> > >> > Firstly, I would like to make sure I can replicate the existing residual >> norm. I tried to do this as: >> > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); >> > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); >> >> I don't understand the lines of code below. I don't see how they could >> compute the norm of the residual. >> >> From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned >> residual with >> >> ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); >> ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); >> ierr = VecDestroy(&resid);CHKERRQ(ierr); >> >> since FGMRES always uses right preconditioning (and hence the >> unpreconditioned residual) this should match the value computed internally >> by FGMRES (subject to roundoff differences) >> >> To exclude certain values in the residual from the norm calculation you >> could write a custom "VecNorm" norm that skipped those entries. >> >> Barry >> >> >> >> >> >> > VecNorm(tmp_sol,NORM_2,&res_tmp); >> > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); >> > res_norm = fabs(res_tmp-res_rhs); >> > >> > But this doesn't match the norm that comes out of UpdateHessenberg. >> > >> > Any ideas on how I can re-create the norm derived from UpdateHessenberg >> as an expansion over the Kyrlov vectors in FGMRES? >> > >> > Cheers, Dave. >> >> From jcrean01 at gmail.com Fri Jul 5 09:26:58 2019 From: jcrean01 at gmail.com (Jared Crean) Date: Fri, 5 Jul 2019 10:26:58 -0400 Subject: [petsc-users] LU Performance In-Reply-To: References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> Message-ID: <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> ??? This is in reply to both David and Barry's emails. ??? I am using the Umfpack that Petsc built (--download-suitesparse=yes was passed to configure), so all the compiler flags and Blas/Lapack libraries are the same.? I used OpenBlas for Blas and Lapack, with multi-threading disabled.? When calling Umfpack directly, the factorization takes about 4 second, compared to 135 seconds spent in MatLUFactorNum when using Umfpack via Petsc. ??? I added a call to umfpack_di_report_control()? (which prints? the Umfpack parameters) to my code, and also added -mat_umfpack_prl 2 to the Petsc options, which should cause Petsc to call the same function just before doing the symbolic factorization (umfpack.c line 245 in Petsc 3.7.6). The output is attached (also with the -ksp_view option).? My code did print the parameters, but Petsc did not, which makes me think MatLUFactorSymbolic_UMFPACK never got called.? For reference, here is how I am invoking the program: ./test -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack -log_view -ksp_view -mat_umfpack_prl 2 > fout_umfpacklu ??? Jared Crean On 7/5/19 4:02 AM, Smith, Barry F. wrote: > When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? > > You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. > > Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). > > Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. > > Barry > > >> On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users wrote: >> >> Hello, >> >> I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. >> >> I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. >> >> >> Jared Crean >> >> -------------- next part -------------- First UMFPack solve reading matrix...finished UMFPACK V5.7.1 (Oct 10, 2014), Control: Matrix entry defined as: double Int (generic integer) defined as: int 0: print level: 2 1: dense row parameter: 0.2 "dense" rows have > max (16, (0.2)*16*sqrt(n_col) entries) 2: dense column parameter: 0.2 "dense" columns have > max (16, (0.2)*16*sqrt(n_row) entries) 3: pivot tolerance: 0.1 4: block size for dense matrix kernels: 32 5: strategy: 0 (auto) 10: ordering: 1 AMD/COLAMD 11: singleton filter: enabled 6: initial allocation ratio: 0.7 7: max iterative refinement steps: 2 13: Q fixed during numerical factorization: 0 (auto) 14: AMD dense row/col parameter: 10 "dense" rows/columns have > max (16, (10)*sqrt(n)) entries Only used if the AMD ordering is used. 15: diagonal pivot tolerance: 0.001 Only used if diagonal pivoting is attempted. 16: scaling: 1 (divide each row by sum of abs. values in each row) 17: frontal matrix allocation ratio: 0.5 18: drop tolerance: 0 19: AMD and COLAMD aggressive absorption: 1 (yes) The following options can only be changed at compile-time: 8: BLAS library used: Fortran BLAS. size of BLAS integer: 4 compiled for ANSI C POSIX C clock_getttime. computer/operating system: Linux size of int: 4 SuiteSparse_long: 8 Int: 4 pointer: 8 double: 8 Entry: 8 (in bytes) symbolic factorization...finished (7.538149e-01 seconds) numeric factorization...finished (3.967724e+00 seconds) backsolve...finished (6.808259e-01 seconds) total elapsed time: 5.456142e+00 seconds First Petsc solve reading matrix...finished preallocating matrix...finished (3.473043e-03 seconds) copying values...finished (7.642679e-01 seconds) KSP solve...KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 12.0718 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=455672, cols=455672 package used to perform factorization: petsc total: nonzeros=182035800, allocated nonzeros=182035800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 113918 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=455672, cols=455672 total: nonzeros=15079424, allocated nonzeros=15079424 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 113918 nodes, limit used is 5 finished (1.395323e+02 seconds) Second UMFPack solve reading matrix...finished UMFPACK V5.7.1 (Oct 10, 2014), Control: Matrix entry defined as: double Int (generic integer) defined as: int 0: print level: 2 1: dense row parameter: 0.2 "dense" rows have > max (16, (0.2)*16*sqrt(n_col) entries) 2: dense column parameter: 0.2 "dense" columns have > max (16, (0.2)*16*sqrt(n_row) entries) 3: pivot tolerance: 0.1 4: block size for dense matrix kernels: 32 5: strategy: 0 (auto) 10: ordering: 1 AMD/COLAMD 11: singleton filter: enabled 6: initial allocation ratio: 0.7 7: max iterative refinement steps: 2 13: Q fixed during numerical factorization: 0 (auto) 14: AMD dense row/col parameter: 10 "dense" rows/columns have > max (16, (10)*sqrt(n)) entries Only used if the AMD ordering is used. 15: diagonal pivot tolerance: 0.001 Only used if diagonal pivoting is attempted. 16: scaling: 1 (divide each row by sum of abs. values in each row) 17: frontal matrix allocation ratio: 0.5 18: drop tolerance: 0 19: AMD and COLAMD aggressive absorption: 1 (yes) The following options can only be changed at compile-time: 8: BLAS library used: Fortran BLAS. size of BLAS integer: 4 compiled for ANSI C POSIX C clock_getttime. computer/operating system: Linux size of int: 4 SuiteSparse_long: 8 Int: 4 pointer: 8 double: 8 Entry: 8 (in bytes) symbolic factorization...finished (7.242250e-01 seconds) numeric factorization...finished (3.962819e+00 seconds) backsolve...finished (6.805091e-01 seconds) total elapsed time: 5.421341e+00 seconds ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./test on a arch-linux2-c-opt named baduk.scorec.rpi.edu with 1 processor, by creanj Fri Jul 5 10:16:56 2019 Using Petsc Release Version 3.7.6, Apr, 24, 2017 Max Max/Min Avg Total Time (sec): 1.519e+02 1.00000 1.519e+02 Objects: 1.200e+01 1.00000 1.200e+01 Flops: 3.612e+11 1.00000 3.612e+11 3.612e+11 Flops/sec: 2.378e+09 1.00000 2.378e+09 2.378e+09 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.5189e+02 100.0% 3.6120e+11 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatSolve 1 1.0 1.8359e-01 1.0 3.64e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1981 MatLUFactorSym 1 1.0 3.4562e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 MatLUFactorNum 1 1.0 1.3577e+02 1.0 3.61e+11 1.0 0.0e+00 0.0e+00 0.0e+00 89100 0 0 0 89100 0 0 0 2658 MatAssemblyBegin 1 1.0 5.0068e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 1 1.0 1.7591e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 1.5704e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.2642e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 9.4652e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 3 1.0 1.8690e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 1 1.0 7.1526e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1 1.0 1.3953e+02 1.0 3.61e+11 1.0 0.0e+00 0.0e+00 0.0e+00 92100 0 0 0 92100 0 0 0 2589 PCSetUp 1 1.0 1.3935e+02 1.0 3.61e+11 1.0 0.0e+00 0.0e+00 0.0e+00 92100 0 0 0 92100 0 0 0 2589 PCApply 1 1.0 1.8360e-01 1.0 3.64e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1980 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 2 2 2372679668 0. Vector 2 2 7293808 0. Krylov Solver 1 1 1160 0. Preconditioner 1 1 992 0. Index Set 5 5 4560600 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 2.38419e-08 #PETSc Option Table entries: -ksp_type preonly -ksp_view -log_view -mat_umfpack_prl 2 -pc_factor_mat_solver_type umfpack -pc_type lu #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-blas-lib=/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2//lib/libopenblas.so --with-lapack-lib=/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2//lib/libopenblas.so --prefix=/opt/scorec/ODL_common/blockade/petsc/3.7.6_opt --download-suitesparse=yes ----------------------------------------- Libraries compiled on Fri Jul 5 08:48:14 2019 on baduk.scorec.rpi.edu Machine characteristics: Linux-3.10.0-957.12.2.el7.x86_64-x86_64-with-redhat-7.6-Maipo Using PETSc directory: /lore/creanj/build/petsc-3.7.6 Using PETSc arch: arch-linux2-c-opt ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -O3 -march=native -mtune=native ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -march=native -mtune=native ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/lore/creanj/build/petsc-3.7.6/arch-linux2-c-opt/include -I/lore/creanj/build/petsc-3.7.6/include -I/lore/creanj/build/petsc-3.7.6/include -I/lore/creanj/build/petsc-3.7.6/arch-linux2-c-opt/include -I/opt/scorec/ODL_common/blockade/petsc/3.7.6_opt/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/lore/creanj/build/petsc-3.7.6/arch-linux2-c-opt/lib -L/lore/creanj/build/petsc-3.7.6/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/opt/scorec/ODL_common/blockade/petsc/3.7.6_opt/lib -L/opt/scorec/ODL_common/blockade/petsc/3.7.6_opt/lib -Wl,-rpath,/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2//lib -L/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2//lib -Wl,-rpath,/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib64 -L/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib64 -Wl,-rpath,/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-diz4f6ieln25ouifyc7ndtqlfksom6nb/lib -L/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-diz4f6ieln25ouifyc7ndtqlfksom6nb/lib -Wl,-rpath,/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -L/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib/gcc/x86_64-pc-linux-gnu/7.3.0 -Wl,-rpath,/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2/lib -L/opt/scorec/ODL_common/baduk/OpenBlas/0.3.2/lib -Wl,-rpath,/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib -L/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib -Wl,-rpath,/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib:/opt/scorec/spack/install/linux-rhel7-x86_64/gcc-rhel7_4.8.5/gcc-7.3.0-bt47fwrzijla4xzdx4o4au45yljqptsk/lib64 -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lopenblas -lX11 -lpthread -lm -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lmpicxx -lstdc++ -lm -ldl -lmpi -lgcc_s -ldl ----------------------------------------- From stefano.zampini at gmail.com Fri Jul 5 09:33:07 2019 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 5 Jul 2019 10:33:07 -0400 Subject: [petsc-users] LU Performance In-Reply-To: <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> Message-ID: Jared, The petsc output shows package used to perform factorization: petsc You are not using umfpack, but the PETSc native LU. You can run with -options_left to see the options that are not processed from the PETSc options database. > On Jul 5, 2019, at 10:26 AM, Jared Crean via petsc-users wrote: > > This is in reply to both David and Barry's emails. > > I am using the Umfpack that Petsc built (--download-suitesparse=yes was passed to configure), so all the compiler flags and Blas/Lapack libraries are the same. I used OpenBlas for Blas and Lapack, with multi-threading disabled. When calling Umfpack directly, the factorization takes about 4 second, compared to 135 seconds spent in MatLUFactorNum when using Umfpack via Petsc. > > I added a call to umfpack_di_report_control() (which prints the Umfpack parameters) to my code, and also added -mat_umfpack_prl 2 to the Petsc options, which should cause Petsc to call the same function just before doing the symbolic factorization (umfpack.c line 245 in Petsc 3.7.6). The output is attached (also with the -ksp_view option). My code did print the parameters, but Petsc did not, which makes me think MatLUFactorSymbolic_UMFPACK never got called. For reference, here is how I am invoking the program: > > ./test -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack -log_view -ksp_view -mat_umfpack_prl 2 > fout_umfpacklu > > Jared Crean > > On 7/5/19 4:02 AM, Smith, Barry F. wrote: >> When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? >> >> You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. >> >> Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). >> >> Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. >> >> Barry >> >> >>> On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users wrote: >>> >>> Hello, >>> >>> I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. >>> >>> I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. >>> >>> >>> Jared Crean >>> >>> > > > From jed at jedbrown.org Fri Jul 5 09:46:56 2019 From: jed at jedbrown.org (Jed Brown) Date: Fri, 05 Jul 2019 08:46:56 -0600 Subject: [petsc-users] LU Performance In-Reply-To: References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> Message-ID: <875zogwnbz.fsf@jedbrown.org> Stefano Zampini via petsc-users writes: > Jared, > > The petsc output shows > > package used to perform factorization: petsc > > You are not using umfpack, but the PETSc native LU. You can run with -options_left to see the options that are not processed from the PETSc options database. -pc_factor_mat_solver_package was used until PETSc-3.9. You're using a rather old (~2016) PETSc with new options. https://www.mcs.anl.gov/petsc/documentation/changes/39.html > > >> On Jul 5, 2019, at 10:26 AM, Jared Crean via petsc-users wrote: >> >> This is in reply to both David and Barry's emails. >> >> I am using the Umfpack that Petsc built (--download-suitesparse=yes was passed to configure), so all the compiler flags and Blas/Lapack libraries are the same. I used OpenBlas for Blas and Lapack, with multi-threading disabled. When calling Umfpack directly, the factorization takes about 4 second, compared to 135 seconds spent in MatLUFactorNum when using Umfpack via Petsc. >> >> I added a call to umfpack_di_report_control() (which prints the Umfpack parameters) to my code, and also added -mat_umfpack_prl 2 to the Petsc options, which should cause Petsc to call the same function just before doing the symbolic factorization (umfpack.c line 245 in Petsc 3.7.6). The output is attached (also with the -ksp_view option). My code did print the parameters, but Petsc did not, which makes me think MatLUFactorSymbolic_UMFPACK never got called. For reference, here is how I am invoking the program: >> >> ./test -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack -log_view -ksp_view -mat_umfpack_prl 2 > fout_umfpacklu >> >> Jared Crean >> >> On 7/5/19 4:02 AM, Smith, Barry F. wrote: >>> When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? >>> >>> You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. >>> >>> Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). >>> >>> Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. >>> >>> Barry >>> >>> >>>> On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users wrote: >>>> >>>> Hello, >>>> >>>> I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. >>>> >>>> I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. >>>> >>>> >>>> Jared Crean >>>> >>>> >> >> >> From shrirang.abhyankar at pnnl.gov Fri Jul 5 09:50:46 2019 From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G) Date: Fri, 5 Jul 2019 14:50:46 +0000 Subject: [petsc-users] LU Performance In-Reply-To: <875zogwnbz.fsf@jedbrown.org> References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> <875zogwnbz.fsf@jedbrown.org> Message-ID: I believe standalone UMFPACK uses appropriate minimum degree (amd) ordering by default provided by package AMD. Use the option -pc_factor_mat_ordering_type amd to use the amd ordering. You can also try quotient minimum degree (qmd) ordering available in PETSc (-pc_factor_mat_ordering_type qmd) Thanks, Shri From: petsc-users on behalf of Jed Brown via petsc-users Reply-To: Jed Brown Date: Friday, July 5, 2019 at 9:48 AM To: Stefano Zampini , Jared Crean Cc: PETSc users list Subject: Re: [petsc-users] LU Performance Stefano Zampini via petsc-users > writes: Jared, The petsc output shows package used to perform factorization: petsc You are not using umfpack, but the PETSc native LU. You can run with -options_left to see the options that are not processed from the PETSc options database. -pc_factor_mat_solver_package was used until PETSc-3.9. You're using a rather old (~2016) PETSc with new options. https://www.mcs.anl.gov/petsc/documentation/changes/39.html On Jul 5, 2019, at 10:26 AM, Jared Crean via petsc-users > wrote: This is in reply to both David and Barry's emails. I am using the Umfpack that Petsc built (--download-suitesparse=yes was passed to configure), so all the compiler flags and Blas/Lapack libraries are the same. I used OpenBlas for Blas and Lapack, with multi-threading disabled. When calling Umfpack directly, the factorization takes about 4 second, compared to 135 seconds spent in MatLUFactorNum when using Umfpack via Petsc. I added a call to umfpack_di_report_control() (which prints the Umfpack parameters) to my code, and also added -mat_umfpack_prl 2 to the Petsc options, which should cause Petsc to call the same function just before doing the symbolic factorization (umfpack.c line 245 in Petsc 3.7.6). The output is attached (also with the -ksp_view option). My code did print the parameters, but Petsc did not, which makes me think MatLUFactorSymbolic_UMFPACK never got called. For reference, here is how I am invoking the program: ./test -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack -log_view -ksp_view -mat_umfpack_prl 2 > fout_umfpacklu Jared Crean On 7/5/19 4:02 AM, Smith, Barry F. wrote: When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. Barry On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users > wrote: Hello, I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. Jared Crean -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrean01 at gmail.com Fri Jul 5 09:53:15 2019 From: jcrean01 at gmail.com (Jared Crean) Date: Fri, 5 Jul 2019 10:53:15 -0400 Subject: [petsc-users] LU Performance In-Reply-To: <875zogwnbz.fsf@jedbrown.org> References: <95399a1a-9b12-2b57-32cb-0a2f23170042@gmail.com> <06f00d9c-1fc4-2395-de66-14c6800965dc@gmail.com> <875zogwnbz.fsf@jedbrown.org> Message-ID: <3d44356d-183b-13ea-255e-bec965b54cb8@gmail.com> ??? Ah, yes, that's it.? I just check the docs that are downloaded with Petsc (rather than the online ones) and it has -pc_factor_mat_solver_package.? I now see similar timings between Petsc and calling Umfpack directly. ??? Thanks, ??? ??? Jared Crean On 7/5/19 10:46 AM, Jed Brown wrote: > Stefano Zampini via petsc-users writes: > >> Jared, >> >> The petsc output shows >> >> package used to perform factorization: petsc >> >> You are not using umfpack, but the PETSc native LU. You can run with -options_left to see the options that are not processed from the PETSc options database. > -pc_factor_mat_solver_package was used until PETSc-3.9. You're using a > rather old (~2016) PETSc with new options. > > https://www.mcs.anl.gov/petsc/documentation/changes/39.html > >> >>> On Jul 5, 2019, at 10:26 AM, Jared Crean via petsc-users wrote: >>> >>> This is in reply to both David and Barry's emails. >>> >>> I am using the Umfpack that Petsc built (--download-suitesparse=yes was passed to configure), so all the compiler flags and Blas/Lapack libraries are the same. I used OpenBlas for Blas and Lapack, with multi-threading disabled. When calling Umfpack directly, the factorization takes about 4 second, compared to 135 seconds spent in MatLUFactorNum when using Umfpack via Petsc. >>> >>> I added a call to umfpack_di_report_control() (which prints the Umfpack parameters) to my code, and also added -mat_umfpack_prl 2 to the Petsc options, which should cause Petsc to call the same function just before doing the symbolic factorization (umfpack.c line 245 in Petsc 3.7.6). The output is attached (also with the -ksp_view option). My code did print the parameters, but Petsc did not, which makes me think MatLUFactorSymbolic_UMFPACK never got called. For reference, here is how I am invoking the program: >>> >>> ./test -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack -log_view -ksp_view -mat_umfpack_prl 2 > fout_umfpacklu >>> >>> Jared Crean >>> >>> On 7/5/19 4:02 AM, Smith, Barry F. wrote: >>>> When you use Umfpack standalone do you use OpenMP threads? When you use umfpack alone do you us thread enabled BLAS/LAPACK? Perhaps OpenBLAS or MKL? >>>> >>>> You can run both cases with -ksp_view and it will print more details indicating indicating the solver used. >>>> >>>> Do you use the same compiler and same options when compiling PETSc and Umfpack standalone. Is the Umfpack standalone time in the numerical factorization much smaller? Perhaps umfpack is using a much better ordering then when used with PETSc (perhaps the default orderings are different). >>>> >>>> Does Umfpack has a routine that tiggers output of the parameters etc it is using? If you can trigger it you might see differences between standalone and not. >>>> >>>> Barry >>>> >>>> >>>>> On Jul 4, 2019, at 4:05 PM, Jared Crean via petsc-users wrote: >>>>> >>>>> Hello, >>>>> >>>>> I am getting very bad performance from the Umfpack LU solver when I use it via Petsc compared to calling Umfpack directly. It takes about 5.5 seconds to factor and solve the matrix with Umfpack, but 140 seconds when I use Petsc with -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type umfpack. >>>>> >>>>> I have attached a minimal example (test.c) that reads a matrix from a file, solves with Umfpack, and then solves with Petsc. The matrix data files are not included because they are about 250 megabytes. I also attached the output of the program with -log_view for -pc_factor_mat_solver_type umfpack (fout_umfpacklu) and -pc_factor_mat_solver_type petsc (fout_petsclu). Both results show nearly all of the time is spent in MatLuFactorNum. The times are very similar, so I am wondering if Petsc is really calling Umfpack or if the Petsc LU solver is getting called in both cases. >>>>> >>>>> >>>>> Jared Crean >>>>> >>>>> >>> >>> From epscodes at gmail.com Fri Jul 5 22:07:20 2019 From: epscodes at gmail.com (Xiangdong) Date: Fri, 5 Jul 2019 23:07:20 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: Hi Mark, After pulling your new commit, I got an error when running ex19 (even without nvprof). Same error with version 3.11.3. But ex19 run fine with old commit 64edb9194a9. Below is a run time log error file. By the way, which version of cuda does PETSc support? Thank you. Xiangdong lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 0 KSP Residual norm 9.158537890035e-02 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1064-g8b74309384 GIT Date: 2019-07-03 10:24:59 -0400 [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 22:57:02 2019 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ aijcusparse.cu [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ aijcusparse.cu [0]PETSC ERROR: #4 MatMult() line 2368 in /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #9 KSPSolve() line 764 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c [0]PETSC ERROR: #11 SNESSolve() line 4430 in /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c [0]PETSC ERROR: #12 main() line 161 in /home/eps/MyCodes/petsctest/extest.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -da_refine 5 [0]PETSC ERROR: -ksp_monitor [0]PETSC ERROR: -mat_type aijcusparse [0]PETSC ERROR: -snes_monitor [0]PETSC ERROR: -snes_view [0]PETSC ERROR: -vec_type cuda [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: > My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. Please > give it a try. > Thanks, > Mark > > On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello everyone, >> >> When I run the ex19 with cuda like this: >> mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor >> -mat_type aijcusparse -vec_type cuda -log_view >> >> it worked fine and produced correct results. >> >> However, when I tried to run this example with nvprof: >> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 >> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda >> -log_view >> >> I got errors like: >> [3]PETSC ERROR: Error in external library >> [3]PETSC ERROR: CUDA error 700 >> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown >> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 >> 2019 >> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 >> --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl >> --with-cuda-dir=/home/eps/MyLocal/cuda >> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install >> --download-hdf5=1 >> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in >> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >> [3]PETSC ERROR: #2 VecSet() line 547 in >> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c >> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in >> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >> [3]PETSC ERROR: #4 VecSetType() line 51 in >> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in >> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >> [3]PETSC ERROR: #6 VecSetType() line 51 in >> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in >> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c >> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in >> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c >> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in >> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c >> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in >> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in >> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c >> [3]PETSC ERROR: #12 SNESSolve() line 4560 in >> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >> [3]PETSC ERROR: #13 main() line 161 in >> /home/eps/MyCodes/petsctest/extest.c >> >> The full run log is attached. >> >> I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA >> Version: 10.1. >> >> I do not know why it is okay without nvprof but crashed with nvprof. Any >> suggestion to fix this? >> >> Thank you. >> >> Best, >> Xiangdong >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 6 12:44:20 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 6 Jul 2019 13:44:20 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: I am not able to reproduce this error. THis is what I get. Are you pulling from mark/gamg-fix-viennacl-rebased ? 13:38 /gpfs/alpine/scratch/adams/geo127$ jsrun -n 1 -c 4 -a 4 -g 1 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 0 KSP Residual norm 9.144944502871e-02 1 KSP Residual norm 2.593759906204e-02 2 KSP Residual norm 1.669815200495e-02 3 KSP Residual norm 1.510777951698e-02 4 KSP Residual norm 1.458401237884e-02 5 KSP Residual norm 1.418635322926e-02 6 KSP Residual norm 1.377436725003e-02 7 KSP Residual norm 1.332236907186e-02 8 KSP Residual norm 1.288602527920e-02 9 KSP Residual norm 1.240018288138e-02 10 KSP Residual norm 1.186798872492e-02 On Fri, Jul 5, 2019 at 11:07 PM Xiangdong wrote: > Hi Mark, > > After pulling your new commit, I got an error when running ex19 (even > without nvprof). Same error with version 3.11.3. But ex19 run fine with old > commit 64edb9194a9. > > Below is a run time log error file. By the way, which version of cuda does > PETSc support? > > Thank you. > > Xiangdong > > lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. > 0 SNES Function norm 1.036007954337e-02 > 0 KSP Residual norm 9.158537890035e-02 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1064-g8b74309384 > GIT Date: 2019-07-03 10:24:59 -0400 > [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 > 22:57:02 2019 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 > --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich > --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 > [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in > /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in > /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ > aijcusparse.cu > [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in > /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ > aijcusparse.cu > [0]PETSC ERROR: #4 MatMult() line 2368 in > /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in > /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in > /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in > /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in > /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #9 KSPSolve() line 764 in > /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in > /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #11 SNESSolve() line 4430 in > /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c > [0]PETSC ERROR: #12 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -da_refine 5 > [0]PETSC ERROR: -ksp_monitor > [0]PETSC ERROR: -mat_type aijcusparse > [0]PETSC ERROR: -snes_monitor > [0]PETSC ERROR: -snes_view > [0]PETSC ERROR: -vec_type cuda > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 > > On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: > >> My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. >> Please give it a try. >> Thanks, >> Mark >> >> On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello everyone, >>> >>> When I run the ex19 with cuda like this: >>> mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor >>> -mat_type aijcusparse -vec_type cuda -log_view >>> >>> it worked fine and produced correct results. >>> >>> However, when I tried to run this example with nvprof: >>> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 >>> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda >>> -log_view >>> >>> I got errors like: >>> [3]PETSC ERROR: Error in external library >>> [3]PETSC ERROR: CUDA error 700 >>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown >>> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 >>> 22:26:01 2019 >>> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 >>> --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl >>> --with-cuda-dir=/home/eps/MyLocal/cuda >>> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install >>> --download-hdf5=1 >>> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in >>> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>> [3]PETSC ERROR: #2 VecSet() line 547 in >>> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c >>> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in >>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>> [3]PETSC ERROR: #4 VecSetType() line 51 in >>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in >>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>> [3]PETSC ERROR: #6 VecSetType() line 51 in >>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in >>> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c >>> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in >>> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c >>> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in >>> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c >>> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in >>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in >>> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c >>> [3]PETSC ERROR: #12 SNESSolve() line 4560 in >>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>> [3]PETSC ERROR: #13 main() line 161 in >>> /home/eps/MyCodes/petsctest/extest.c >>> >>> The full run log is attached. >>> >>> I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA >>> Version: 10.1. >>> >>> I do not know why it is okay without nvprof but crashed with nvprof. Any >>> suggestion to fix this? >>> >>> Thank you. >>> >>> Best, >>> Xiangdong >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Sat Jul 6 15:07:32 2019 From: epscodes at gmail.com (Xiangdong) Date: Sat, 6 Jul 2019 16:07:32 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: Hi Mark, Yes, I am using your branch (mark/gamg-fix-viennacl-rebased) with Petsc Development GIT revision: v3.11.3-1064-g8b74309384. I compiled PETSc with CUDA 10.1. Which version of CUDA are you using? Thanks. Xiangdong On Sat, Jul 6, 2019 at 1:43 PM Mark Adams wrote: > I am not able to reproduce this error. THis is what I get. Are you pulling > from mark/gamg-fix-viennacl-rebased ? > > 13:38 /gpfs/alpine/scratch/adams/geo127$ jsrun -n 1 -c 4 -a 4 -g 1 ./ex19 > -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse > -vec_type cuda -log_view > lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. > 0 SNES Function norm 1.036007954337e-02 > 0 KSP Residual norm 9.144944502871e-02 > 1 KSP Residual norm 2.593759906204e-02 > 2 KSP Residual norm 1.669815200495e-02 > 3 KSP Residual norm 1.510777951698e-02 > 4 KSP Residual norm 1.458401237884e-02 > 5 KSP Residual norm 1.418635322926e-02 > 6 KSP Residual norm 1.377436725003e-02 > 7 KSP Residual norm 1.332236907186e-02 > 8 KSP Residual norm 1.288602527920e-02 > 9 KSP Residual norm 1.240018288138e-02 > 10 KSP Residual norm 1.186798872492e-02 > > > On Fri, Jul 5, 2019 at 11:07 PM Xiangdong wrote: > >> Hi Mark, >> >> After pulling your new commit, I got an error when running ex19 (even >> without nvprof). Same error with version 3.11.3. But ex19 run fine with old >> commit 64edb9194a9. >> >> Below is a run time log error file. By the way, which version of cuda >> does PETSc support? >> >> Thank you. >> >> Xiangdong >> >> lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. >> 0 SNES Function norm 1.036007954337e-02 >> 0 KSP Residual norm 9.158537890035e-02 >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1064-g8b74309384 >> GIT Date: 2019-07-03 10:24:59 -0400 >> [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 >> 22:57:02 2019 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 >> --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 >> [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in >> /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >> [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in >> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >> aijcusparse.cu >> [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in >> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >> aijcusparse.cu >> [0]PETSC ERROR: #4 MatMult() line 2368 in >> /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c >> [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in >> /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in >> /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h >> [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in >> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in >> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #9 KSPSolve() line 764 in >> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in >> /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #11 SNESSolve() line 4430 in >> /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c >> [0]PETSC ERROR: #12 main() line 161 in >> /home/eps/MyCodes/petsctest/extest.c >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -da_refine 5 >> [0]PETSC ERROR: -ksp_monitor >> [0]PETSC ERROR: -mat_type aijcusparse >> [0]PETSC ERROR: -snes_monitor >> [0]PETSC ERROR: -snes_view >> [0]PETSC ERROR: -vec_type cuda >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 >> >> On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: >> >>> My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. >>> Please give it a try. >>> Thanks, >>> Mark >>> >>> On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello everyone, >>>> >>>> When I run the ex19 with cuda like this: >>>> mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor >>>> -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view >>>> >>>> it worked fine and produced correct results. >>>> >>>> However, when I tried to run this example with nvprof: >>>> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 >>>> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda >>>> -log_view >>>> >>>> I got errors like: >>>> [3]PETSC ERROR: Error in external library >>>> [3]PETSC ERROR: CUDA error 700 >>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown >>>> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 >>>> 22:26:01 2019 >>>> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt >>>> --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl >>>> --with-cuda-dir=/home/eps/MyLocal/cuda >>>> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install >>>> --download-hdf5=1 >>>> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>>> [3]PETSC ERROR: #2 VecSet() line 547 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c >>>> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>> [3]PETSC ERROR: #4 VecSetType() line 51 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>> [3]PETSC ERROR: #6 VecSetType() line 51 in >>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in >>>> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c >>>> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in >>>> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c >>>> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in >>>> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c >>>> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in >>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in >>>> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c >>>> [3]PETSC ERROR: #12 SNESSolve() line 4560 in >>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>> [3]PETSC ERROR: #13 main() line 161 in >>>> /home/eps/MyCodes/petsctest/extest.c >>>> >>>> The full run log is attached. >>>> >>>> I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA >>>> Version: 10.1. >>>> >>>> I do not know why it is okay without nvprof but crashed with nvprof. >>>> Any suggestion to fix this? >>>> >>>> Thank you. >>>> >>>> Best, >>>> Xiangdong >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 6 16:52:12 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 6 Jul 2019 17:52:12 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: I'm using CUDA 10.1.105 and this looks like a logic bug and not a CUDA problem. I added two MatSetType calls that I thought were the right thing to do. Your error is a type problem and test with DMPlex so the matrix creation goes through a different code path. You could try removing them and see if it works. That is really all there is in this branch at this point other than some ViennaCL stuff. I would verify that a fresh pull from master works. I would also reconfigure. I have just pulled from master and pushed so you can do a fresh pull. On Sat, Jul 6, 2019 at 4:07 PM Xiangdong wrote: > Hi Mark, > > Yes, I am using your branch (mark/gamg-fix-viennacl-rebased) with Petsc > Development GIT revision: v3.11.3-1064-g8b74309384. I compiled PETSc with > CUDA 10.1. > > Which version of CUDA are you using? > > Thanks. > > Xiangdong > > On Sat, Jul 6, 2019 at 1:43 PM Mark Adams wrote: > >> I am not able to reproduce this error. THis is what I get. Are you >> pulling from mark/gamg-fix-viennacl-rebased ? >> >> 13:38 /gpfs/alpine/scratch/adams/geo127$ jsrun -n 1 -c 4 -a 4 -g 1 >> ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type >> aijcusparse -vec_type cuda -log_view >> lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. >> 0 SNES Function norm 1.036007954337e-02 >> 0 KSP Residual norm 9.144944502871e-02 >> 1 KSP Residual norm 2.593759906204e-02 >> 2 KSP Residual norm 1.669815200495e-02 >> 3 KSP Residual norm 1.510777951698e-02 >> 4 KSP Residual norm 1.458401237884e-02 >> 5 KSP Residual norm 1.418635322926e-02 >> 6 KSP Residual norm 1.377436725003e-02 >> 7 KSP Residual norm 1.332236907186e-02 >> 8 KSP Residual norm 1.288602527920e-02 >> 9 KSP Residual norm 1.240018288138e-02 >> 10 KSP Residual norm 1.186798872492e-02 >> >> >> On Fri, Jul 5, 2019 at 11:07 PM Xiangdong wrote: >> >>> Hi Mark, >>> >>> After pulling your new commit, I got an error when running ex19 (even >>> without nvprof). Same error with version 3.11.3. But ex19 run fine with old >>> commit 64edb9194a9. >>> >>> Below is a run time log error file. By the way, which version of cuda >>> does PETSc support? >>> >>> Thank you. >>> >>> Xiangdong >>> >>> lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. >>> 0 SNES Function norm 1.036007954337e-02 >>> 0 KSP Residual norm 9.158537890035e-02 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1064-g8b74309384 >>> GIT Date: 2019-07-03 10:24:59 -0400 >>> [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 >>> 22:57:02 2019 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 >>> --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 >>> [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in >>> /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>> [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in >>> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >>> aijcusparse.cu >>> [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in >>> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >>> aijcusparse.cu >>> [0]PETSC ERROR: #4 MatMult() line 2368 in >>> /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in >>> /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in >>> /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h >>> [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in >>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in >>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: #9 KSPSolve() line 764 in >>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in >>> /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: #11 SNESSolve() line 4430 in >>> /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c >>> [0]PETSC ERROR: #12 main() line 161 in >>> /home/eps/MyCodes/petsctest/extest.c >>> [0]PETSC ERROR: PETSc Option Table entries: >>> [0]PETSC ERROR: -da_refine 5 >>> [0]PETSC ERROR: -ksp_monitor >>> [0]PETSC ERROR: -mat_type aijcusparse >>> [0]PETSC ERROR: -snes_monitor >>> [0]PETSC ERROR: -snes_view >>> [0]PETSC ERROR: -vec_type cuda >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>> error message to petsc-maint at mcs.anl.gov---------- >>> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 >>> >>> On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: >>> >>>> My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. >>>> Please give it a try. >>>> Thanks, >>>> Mark >>>> >>>> On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> When I run the ex19 with cuda like this: >>>>> mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor >>>>> -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view >>>>> >>>>> it worked fine and produced correct results. >>>>> >>>>> However, when I tried to run this example with nvprof: >>>>> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 >>>>> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda >>>>> -log_view >>>>> >>>>> I got errors like: >>>>> [3]PETSC ERROR: Error in external library >>>>> [3]PETSC ERROR: CUDA error 700 >>>>> [3]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown >>>>> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 >>>>> 22:26:01 2019 >>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt >>>>> --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>>>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl >>>>> --with-cuda-dir=/home/eps/MyLocal/cuda >>>>> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install >>>>> --download-hdf5=1 >>>>> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>>>> [3]PETSC ERROR: #2 VecSet() line 547 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c >>>>> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>>> [3]PETSC ERROR: #4 VecSetType() line 51 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>>> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>>> [3]PETSC ERROR: #6 VecSetType() line 51 in >>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>>> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in >>>>> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c >>>>> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in >>>>> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c >>>>> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in >>>>> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c >>>>> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in >>>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>>> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in >>>>> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c >>>>> [3]PETSC ERROR: #12 SNESSolve() line 4560 in >>>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>>> [3]PETSC ERROR: #13 main() line 161 in >>>>> /home/eps/MyCodes/petsctest/extest.c >>>>> >>>>> The full run log is attached. >>>>> >>>>> I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA >>>>> Version: 10.1. >>>>> >>>>> I do not know why it is okay without nvprof but crashed with nvprof. >>>>> Any suggestion to fix this? >>>>> >>>>> Thank you. >>>>> >>>>> Best, >>>>> Xiangdong >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From davelee2804 at gmail.com Sat Jul 6 18:03:12 2019 From: davelee2804 at gmail.com (Dave Lee) Date: Sun, 7 Jul 2019 09:03:12 +1000 Subject: [petsc-users] Computing residual norm in KSPFGMRESCycle() In-Reply-To: <87bly8ws5m.fsf@jedbrown.org> References: <8E66A833-45CC-449A-B2F2-6C6D61DD5295@anl.gov> <87bly8ws5m.fsf@jedbrown.org> Message-ID: Thanks for the tip Jed, I will look into this. I'm currently trying to figure out if there's any way I can get away with using the existing L2 norm, and if not GCR sounds like a good alternative to pursue. Cheers, Dave. On Fri, Jul 5, 2019 at 11:02 PM Jed Brown wrote: > Dave, have you considered using GCR instead of FGMRES? It's a flexible > method that is equivalent in many circumstances, but provides the > residual and solution at each iteration without needing to "build" it. > > Dave Lee via petsc-users writes: > > > Hi Matt and Barry, > > > > thanks for the good ideas. > > > > I wasn't aware of the function KSPSetConvergenceTest(), thanks for > alerting > > me to this Matt. Unfortunately I really do need to exclude a couple of > > specific degrees of freedom from the norm, and as such I want to make > sure > > I can replicate the norm as computed within UpdateHessenberg() before I > > replace this. > > > > Apologies Barry, there was a stupid bug in that code, it should have > read: > > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > > KSPFGMRESBuildSoln(fgmres->nrs,ksp->vec_sol,tmp_sol,ksp,loc_it+1); > > VecAXPY(tmp_sol,-1.0,ksp->vec_rhs); > > VecNorm(tmp_sol,NORM_2,&res_tmp); > > > > Which does not give the same result as the norm for UpdateHessenberg(). > > Unfortunately I want to avoid using > > KSPBuildResidual(ksp,NULL,NULL,&resid); > > as this calls KSP_MatMult(), which will in turn call my snes residual > > assembly routine. This routine in turn calls a Navier Stokes solver to > run > > for a fixed (long) time period, and so is expensive. > > > > Replacing VecNorm() with my own version is indeed what I'm doing > elsewhere, > > I'm just not sure of how to replace the UpdateHessenberg() norm right > now. > > > > Cheers, Dave. > > > > On Fri, Jul 5, 2019 at 12:59 PM Smith, Barry F. > wrote: > > > >> > >> > >> > On Jul 4, 2019, at 7:29 PM, Dave Lee via petsc-users < > >> petsc-users at mcs.anl.gov> wrote: > >> > > >> > Hi PETSc, > >> > > >> > I have a problem for which I need to exclude certain degrees of > freedom > >> from the evaluation of the norm as used to monitor the residual in > FGMRES > >> (don't ask!). Therefore I would like to replace the existing norm > >> evaluation with my own version. > >> > > >> > Currently this is done within: > >> > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm) > >> > as: > >> > *res = PetscAbsScalar(*RS(it+1)); > >> > > >> > Firstly, I would like to make sure I can replicate the existing > residual > >> norm. I tried to do this as: > >> > KSPFGMRESUpdateHessenberg(ksp,loc_it,hapend,&res_norm); > >> > KSPFGMRESBuildSoln(fgmres->nrs,ksp >vec_sol,tmp_sol,ksp,loc_it); > >> > >> I don't understand the lines of code below. I don't see how they > could > >> compute the norm of the residual. > >> > >> From KSPMonitorTrueResidualNorm() you can compute the unpreconditioned > >> residual with > >> > >> ierr = KSPBuildResidual(ksp,NULL,NULL,&resid);CHKERRQ(ierr); > >> ierr = VecNorm(resid,NORM_2,&truenorm);CHKERRQ(ierr); > >> ierr = VecDestroy(&resid);CHKERRQ(ierr); > >> > >> since FGMRES always uses right preconditioning (and hence the > >> unpreconditioned residual) this should match the value computed > internally > >> by FGMRES (subject to roundoff differences) > >> > >> To exclude certain values in the residual from the norm calculation you > >> could write a custom "VecNorm" norm that skipped those entries. > >> > >> Barry > >> > >> > >> > >> > >> > >> > VecNorm(tmp_sol,NORM_2,&res_tmp); > >> > VecNorm(ksp->vec_rhs,NORM_2,&res_rhs); > >> > res_norm = fabs(res_tmp-res_rhs); > >> > > >> > But this doesn't match the norm that comes out of UpdateHessenberg. > >> > > >> > Any ideas on how I can re-create the norm derived from > UpdateHessenberg > >> as an expansion over the Kyrlov vectors in FGMRES? > >> > > >> > Cheers, Dave. > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Sat Jul 6 22:35:34 2019 From: epscodes at gmail.com (Xiangdong) Date: Sat, 6 Jul 2019 23:35:34 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: Hi Mark, For your latest commit 9ab9e86e1e, I can get the normal run working by either commenting out all PetscCheckTypeNames in veccuda2.cu or use "-dm_vec_type cuda" instead of "-vec_type cuda". However, even with your latest commit, I still got an error when using nvprof (different lines reported). I checked that I did use same cuda driver and library (cuda_10.1.168_418.67_linux). eps at new:~/MyCodes/petsctest$ nvprof --profile-child-processes mpiexec -np 1 ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda ==8902== NVPROF is profiling process 8902, command: ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: CUDA error 700 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1103-g9ab9e86e1e GIT Date: 2019-07-06 17:40:27 -0400 [0]PETSC ERROR: ./extest on a arch-debug-origin named new by eps Sat Jul 6 23:24:58 2019 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug-origin --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-cuda=1 --download-fblaslapack [0]PETSC ERROR: #1 VecCUDACopyFromGPU() line 131 in /home/eps/MyLocal/petsc-latest/src/vec/vec/impls/seq/seqcuda/veccuda2.cu [0]PETSC ERROR: #2 VecGetArray() line 1556 in /home/eps/MyLocal/petsc-latest/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #3 VecGetArray2d() line 2185 in /home/eps/MyLocal/petsc-latest/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #4 DMDAVecGetArray() line 68 in /home/eps/MyLocal/petsc-latest/src/dm/impls/da/dagetarray.c [0]PETSC ERROR: #5 FormInitialGuess() line 223 in /home/eps/MyCodes/petsctest/extest.c [0]PETSC ERROR: #6 main() line 159 in /home/eps/MyCodes/petsctest/extest.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -da_refine 5 [0]PETSC ERROR: -dm_vec_type cuda [0]PETSC ERROR: -ksp_monitor [0]PETSC ERROR: -mat_type aijcusparse [0]PETSC ERROR: -snes_monitor [0]PETSC ERROR: -snes_view [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 ==8902== Profiling application: ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda ==8902== Profiling result: No kernels were profiled. No API activities were profiled. ==8902== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. ======== Error: Application returned non-zero code 76 Thank you. Best, Xiangdong On Sat, Jul 6, 2019 at 5:51 PM Mark Adams wrote: > I'm using CUDA 10.1.105 and this looks like a logic bug and not a CUDA > problem. > > I added two MatSetType calls that I thought were the right thing to do. > Your error is a type problem and test with DMPlex so the matrix creation > goes through a different code path. > > You could try removing them and see if it works. > > That is really all there is in this branch at this point other than some > ViennaCL stuff. > > I would verify that a fresh pull from master works. I would also > reconfigure. I have just pulled from master and pushed so you can do a > fresh pull. > > > On Sat, Jul 6, 2019 at 4:07 PM Xiangdong wrote: > >> Hi Mark, >> >> Yes, I am using your branch (mark/gamg-fix-viennacl-rebased) with Petsc >> Development GIT revision: v3.11.3-1064-g8b74309384. I compiled PETSc with >> CUDA 10.1. >> >> Which version of CUDA are you using? >> >> Thanks. >> >> Xiangdong >> >> On Sat, Jul 6, 2019 at 1:43 PM Mark Adams wrote: >> >>> I am not able to reproduce this error. THis is what I get. Are you >>> pulling from mark/gamg-fix-viennacl-rebased ? >>> >>> 13:38 /gpfs/alpine/scratch/adams/geo127$ jsrun -n 1 -c 4 -a 4 -g 1 >>> ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type >>> aijcusparse -vec_type cuda -log_view >>> lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. >>> 0 SNES Function norm 1.036007954337e-02 >>> 0 KSP Residual norm 9.144944502871e-02 >>> 1 KSP Residual norm 2.593759906204e-02 >>> 2 KSP Residual norm 1.669815200495e-02 >>> 3 KSP Residual norm 1.510777951698e-02 >>> 4 KSP Residual norm 1.458401237884e-02 >>> 5 KSP Residual norm 1.418635322926e-02 >>> 6 KSP Residual norm 1.377436725003e-02 >>> 7 KSP Residual norm 1.332236907186e-02 >>> 8 KSP Residual norm 1.288602527920e-02 >>> 9 KSP Residual norm 1.240018288138e-02 >>> 10 KSP Residual norm 1.186798872492e-02 >>> >>> >>> On Fri, Jul 5, 2019 at 11:07 PM Xiangdong wrote: >>> >>>> Hi Mark, >>>> >>>> After pulling your new commit, I got an error when running ex19 (even >>>> without nvprof). Same error with version 3.11.3. But ex19 run fine with old >>>> commit 64edb9194a9. >>>> >>>> Below is a run time log error file. By the way, which version of cuda >>>> does PETSc support? >>>> >>>> Thank you. >>>> >>>> Xiangdong >>>> >>>> lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. >>>> 0 SNES Function norm 1.036007954337e-02 >>>> 0 KSP Residual norm 9.158537890035e-02 >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda >>>> [0]PETSC ERROR: See >>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>> shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.11.3-1064-g8b74309384 GIT Date: 2019-07-03 10:24:59 -0400 >>>> [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 >>>> 22:57:02 2019 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 >>>> --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 >>>> [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in >>>> /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>>> [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in >>>> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >>>> aijcusparse.cu >>>> [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in >>>> /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/ >>>> aijcusparse.cu >>>> [0]PETSC ERROR: #4 MatMult() line 2368 in >>>> /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in >>>> /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in >>>> /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h >>>> [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in >>>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in >>>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: #9 KSPSolve() line 764 in >>>> /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in >>>> /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c >>>> [0]PETSC ERROR: #11 SNESSolve() line 4430 in >>>> /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #12 main() line 161 in >>>> /home/eps/MyCodes/petsctest/extest.c >>>> [0]PETSC ERROR: PETSc Option Table entries: >>>> [0]PETSC ERROR: -da_refine 5 >>>> [0]PETSC ERROR: -ksp_monitor >>>> [0]PETSC ERROR: -mat_type aijcusparse >>>> [0]PETSC ERROR: -snes_monitor >>>> [0]PETSC ERROR: -snes_view >>>> [0]PETSC ERROR: -vec_type cuda >>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>>> error message to petsc-maint at mcs.anl.gov---------- >>>> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 >>>> >>>> On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: >>>> >>>>> My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. >>>>> Please give it a try. >>>>> Thanks, >>>>> Mark >>>>> >>>>> On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users < >>>>> petsc-users at mcs.anl.gov> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> When I run the ex19 with cuda like this: >>>>>> mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor >>>>>> -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view >>>>>> >>>>>> it worked fine and produced correct results. >>>>>> >>>>>> However, when I tried to run this example with nvprof: >>>>>> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 >>>>>> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda >>>>>> -log_view >>>>>> >>>>>> I got errors like: >>>>>> [3]PETSC ERROR: Error in external library >>>>>> [3]PETSC ERROR: CUDA error 700 >>>>>> [3]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>> shooting. >>>>>> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown >>>>>> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 >>>>>> 22:26:01 2019 >>>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt >>>>>> --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich >>>>>> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl >>>>>> --with-cuda-dir=/home/eps/MyLocal/cuda >>>>>> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install >>>>>> --download-hdf5=1 >>>>>> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu >>>>>> [3]PETSC ERROR: #2 VecSet() line 547 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c >>>>>> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>>>> [3]PETSC ERROR: #4 VecSetType() line 51 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>>>> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu >>>>>> [3]PETSC ERROR: #6 VecSetType() line 51 in >>>>>> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c >>>>>> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in >>>>>> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c >>>>>> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in >>>>>> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c >>>>>> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in >>>>>> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c >>>>>> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in >>>>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>>>> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in >>>>>> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c >>>>>> [3]PETSC ERROR: #12 SNESSolve() line 4560 in >>>>>> /home/eps/MyLocal/petsc/src/snes/interface/snes.c >>>>>> [3]PETSC ERROR: #13 main() line 161 in >>>>>> /home/eps/MyCodes/petsctest/extest.c >>>>>> >>>>>> The full run log is attached. >>>>>> >>>>>> I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA >>>>>> Version: 10.1. >>>>>> >>>>>> I do not know why it is okay without nvprof but crashed with nvprof. >>>>>> Any suggestion to fix this? >>>>>> >>>>>> Thank you. >>>>>> >>>>>> Best, >>>>>> Xiangdong >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jul 6 23:29:40 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 7 Jul 2019 04:29:40 +0000 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: err = cudaMemcpy(((Vec_Seq*)v->data)->array,varray,v->map->n*sizeof(PetscScalar),cudaMemcpyDeviceToHost);CHKERRCUDA(err); can you run in the debugger at the same time as using this tool (Seems like you should be able to)? You could then check the value of v->map->n and ((Vec_Seq*)v->data)->array and varray to see if they are all reasonable. Is it, for example, choking on v->map->n being zero or are either of the arrays garbage values or all three? For example if n is zero it may still be checking if the two arrays are reasonable even though they may not be since no data needs to be copied. Barry If cudaMemcpy() cannot handle n of 0 gracefully for example we may need to protect it and only call when n > 0 > On Jul 6, 2019, at 10:35 PM, Xiangdong via petsc-users wrote: > > Hi Mark, > > For your latest commit 9ab9e86e1e, I can get the normal run working by either commenting out all PetscCheckTypeNames in veccuda2.cu or use "-dm_vec_type cuda" instead of "-vec_type cuda". > > However, even with your latest commit, I still got an error when using nvprof (different lines reported). I checked that I did use same cuda driver and library (cuda_10.1.168_418.67_linux). > > eps at new:~/MyCodes/petsctest$ nvprof --profile-child-processes mpiexec -np 1 ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda > ==8902== NVPROF is profiling process 8902, command: ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda > lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: CUDA error 700 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1103-g9ab9e86e1e GIT Date: 2019-07-06 17:40:27 -0400 > [0]PETSC ERROR: ./extest on a arch-debug-origin named new by eps Sat Jul 6 23:24:58 2019 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug-origin --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-cuda=1 --download-fblaslapack > [0]PETSC ERROR: #1 VecCUDACopyFromGPU() line 131 in /home/eps/MyLocal/petsc-latest/src/vec/vec/impls/seq/seqcuda/veccuda2.c } > [0]PETSC ERROR: #2 VecGetArray() line 1556 in /home/eps/MyLocal/petsc-latest/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #3 VecGetArray2d() line 2185 in /home/eps/MyLocal/petsc-latest/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #4 DMDAVecGetArray() line 68 in /home/eps/MyLocal/petsc-latest/src/dm/impls/da/dagetarray.c > [0]PETSC ERROR: #5 FormInitialGuess() line 223 in /home/eps/MyCodes/petsctest/extest.c > [0]PETSC ERROR: #6 main() line 159 in /home/eps/MyCodes/petsctest/extest.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -da_refine 5 > [0]PETSC ERROR: -dm_vec_type cuda > [0]PETSC ERROR: -ksp_monitor > [0]PETSC ERROR: -mat_type aijcusparse > [0]PETSC ERROR: -snes_monitor > [0]PETSC ERROR: -snes_view > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 > ==8902== Profiling application: ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -dm_vec_type cuda > ==8902== Profiling result: > No kernels were profiled. > No API activities were profiled. > ==8902== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data. > ======== Error: Application returned non-zero code 76 > > Thank you. > > Best, > Xiangdong > > On Sat, Jul 6, 2019 at 5:51 PM Mark Adams wrote: > I'm using CUDA 10.1.105 and this looks like a logic bug and not a CUDA problem. > > I added two MatSetType calls that I thought were the right thing to do. Your error is a type problem and test with DMPlex so the matrix creation goes through a different code path. > > You could try removing them and see if it works. > > That is really all there is in this branch at this point other than some ViennaCL stuff. > > I would verify that a fresh pull from master works. I would also reconfigure. I have just pulled from master and pushed so you can do a fresh pull. > > > On Sat, Jul 6, 2019 at 4:07 PM Xiangdong wrote: > Hi Mark, > > Yes, I am using your branch (mark/gamg-fix-viennacl-rebased) with Petsc Development GIT revision: v3.11.3-1064-g8b74309384. I compiled PETSc with CUDA 10.1. > > Which version of CUDA are you using? > > Thanks. > > Xiangdong > > On Sat, Jul 6, 2019 at 1:43 PM Mark Adams wrote: > I am not able to reproduce this error. THis is what I get. Are you pulling from mark/gamg-fix-viennacl-rebased ? > > 13:38 /gpfs/alpine/scratch/adams/geo127$ jsrun -n 1 -c 4 -a 4 -g 1 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view > lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. > 0 SNES Function norm 1.036007954337e-02 > 0 KSP Residual norm 9.144944502871e-02 > 1 KSP Residual norm 2.593759906204e-02 > 2 KSP Residual norm 1.669815200495e-02 > 3 KSP Residual norm 1.510777951698e-02 > 4 KSP Residual norm 1.458401237884e-02 > 5 KSP Residual norm 1.418635322926e-02 > 6 KSP Residual norm 1.377436725003e-02 > 7 KSP Residual norm 1.332236907186e-02 > 8 KSP Residual norm 1.288602527920e-02 > 9 KSP Residual norm 1.240018288138e-02 > 10 KSP Residual norm 1.186798872492e-02 > > > On Fri, Jul 5, 2019 at 11:07 PM Xiangdong wrote: > Hi Mark, > > After pulling your new commit, I got an error when running ex19 (even without nvprof). Same error with version 3.11.3. But ex19 run fine with old commit 64edb9194a9. > > Below is a run time log error file. By the way, which version of cuda does PETSc support? > > Thank you. > > Xiangdong > > lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. > 0 SNES Function norm 1.036007954337e-02 > 0 KSP Residual norm 9.158537890035e-02 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1064-g8b74309384 GIT Date: 2019-07-03 10:24:59 -0400 > [0]PETSC ERROR: ./extest on a arch-debug2 named new by eps Fri Jul 5 22:57:02 2019 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-debug2 --with-debugging=1 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda=1 > [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 1304 in /home/eps/MyLocal/petsc-mark/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1415 in /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu > [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1347 in /home/eps/MyLocal/petsc-mark/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu > [0]PETSC ERROR: #4 MatMult() line 2368 in /home/eps/MyLocal/petsc-mark/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCApplyBAorAB() line 662 in /home/eps/MyLocal/petsc-mark/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() line 309 in /home/eps/MyLocal/petsc-mark/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #7 KSPGMRESCycle() line 152 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 237 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #9 KSPSolve() line 764 in /home/eps/MyLocal/petsc-mark/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 225 in /home/eps/MyLocal/petsc-mark/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #11 SNESSolve() line 4430 in /home/eps/MyLocal/petsc-mark/src/snes/interface/snes.c > [0]PETSC ERROR: #12 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -da_refine 5 > [0]PETSC ERROR: -ksp_monitor > [0]PETSC ERROR: -mat_type aijcusparse > [0]PETSC ERROR: -snes_monitor > [0]PETSC ERROR: -snes_view > [0]PETSC ERROR: -vec_type cuda > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 > > On Wed, Jul 3, 2019 at 10:27 AM Mark Adams wrote: > My branch mark/gamg-fix-viennacl-rebased fixes this problem for me. Please give it a try. > Thanks, > Mark > > On Tue, Jul 2, 2019 at 10:41 PM Xiangdong via petsc-users wrote: > Hello everyone, > > When I run the ex19 with cuda like this: > mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view > > it worked fine and produced correct results. > > However, when I tried to run this example with nvprof: > nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view > > I got errors like: > [3]PETSC ERROR: Error in external library > [3]PETSC ERROR: CUDA error 700 > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown > [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul 2 22:26:01 2019 > [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda-dir=/home/eps/MyLocal/cuda --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install --download-hdf5=1 > [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu > [3]PETSC ERROR: #2 VecSet() line 547 in /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c > [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #4 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu > [3]PETSC ERROR: #6 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c > [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in /home/eps/MyLocal/petsc/src/mat/interface/matrix.c > [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c > [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c > [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c > [3]PETSC ERROR: #12 SNESSolve() line 4560 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c > [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c > > The full run log is attached. > > I am using NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1. > > I do not know why it is okay without nvprof but crashed with nvprof. Any suggestion to fix this? > > Thank you. > > Best, > Xiangdong > From mfadams at lbl.gov Sun Jul 7 18:05:07 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 7 Jul 2019 19:05:07 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: > > > > [0]PETSC ERROR: -dm_vec_type cuda > [0]PETSC ERROR: -ksp_monitor > [0]PETSC ERROR: -mat_type aijcusparse > You might want -*dm_*mat_type aijcusparse -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Sun Jul 7 20:37:15 2019 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Mon, 8 Jul 2019 13:37:15 +1200 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> Message-ID: <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> hi Matt, On 29/06/19 12:07 AM, Matthew Knepley wrote: > > Okay, I ran through this. I am attaching my C version which was easier > for me to play with in the debugger, but it should not be > hard to put the few lines into yours. The problem comes from using > overlap in the redistribution. You can see this by running > > ? mpiexec -n 2 ./distgeom2 -overlap 1 -overlap2 0 > > which works fine. If overlap2 is 1, then the face data is shifted. Yes, that's the same thing I found. > Something in DistributeField() does not know that we changed > the topology after we distributed by adding overlap cells. This is > strange since DMPlexMigrate() works on the coordinates for > the overlapped mesh, however they always start at 0 in the vector, so > it must be that some offset is missing. I am looking for it now. It looked to me like the problem might lie not in DMPlexDistributeField() but in DMPlexDistribute(), because if you view the redistribution SF coming out of the second call to DMPlexDistribute() it looks wrong. DMPlexDistributeField() appeared to be doing what it was told, but the SF it was given didn't look right to me. - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 7 20:48:59 2019 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 7 Jul 2019 20:48:59 -0500 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> Message-ID: On Sun, Jul 7, 2019 at 8:37 PM Adrian Croucher wrote: > hi Matt, > On 29/06/19 12:07 AM, Matthew Knepley wrote: > > > Okay, I ran through this. I am attaching my C version which was easier for > me to play with in the debugger, but it should not be > hard to put the few lines into yours. The problem comes from using overlap > in the redistribution. You can see this by running > > mpiexec -n 2 ./distgeom2 -overlap 1 -overlap2 0 > > which works fine. If overlap2 is 1, then the face data is shifted. > > > Yes, that's the same thing I found. > > Something in DistributeField() does not know that we changed > the topology after we distributed by adding overlap cells. This is strange > since DMPlexMigrate() works on the coordinates for > the overlapped mesh, however they always start at 0 in the vector, so it > must be that some offset is missing. I am looking for it now. > > > It looked to me like the problem might lie not in DMPlexDistributeField() > but in DMPlexDistribute(), because if you view the redistribution SF coming > out of the second call to DMPlexDistribute() it looks wrong. > > DMPlexDistributeField() appeared to be doing what it was told, but the SF > it was given didn't look right to me. > > True. I fixed it, but I am just getting the PR in order. Should have it tomorrow morning. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Sun Jul 7 22:01:06 2019 From: epscodes at gmail.com (Xiangdong) Date: Sun, 7 Jul 2019 23:01:06 -0400 Subject: [petsc-users] snes/ex19 issue with nvprof In-Reply-To: References: Message-ID: It turned out to be the issue of cuda version. I installed the 10.1.105 and it is working fine now. The one I had issue with is the latest 10.1.168. Thank you. Xiangdong On Sun, Jul 7, 2019 at 7:04 PM Mark Adams wrote: > >> >> [0]PETSC ERROR: -dm_vec_type cuda >> [0]PETSC ERROR: -ksp_monitor >> [0]PETSC ERROR: -mat_type aijcusparse >> > > You might want -*dm_*mat_type aijcusparse > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Jul 8 18:53:02 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 8 Jul 2019 16:53:02 -0700 Subject: [petsc-users] Implementing TS routine In-Reply-To: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> Message-ID: Hi Zhang, Thanks to your help i have implemented the TS routine for my temperature DMDA array, and it works correctly including in MPI. The breakthrough for me was realizing that TS steps and max time steps are all set up once, and used/advanced all at once when invoked with TSSolution. So what i did was to add all of the TSCreate / initialization into the main loop so i am basically creating and destroying the TS objects every time step. I hope the overhead time is not to bad when i scale to bigger problems. Next step would be to apply the same to advance the velocities, but problem now is that they are in a 4D dmda array of 3 degrees of freedom, any suggestions on how to implement this? does TS support multiple degrees of freedom arrays? Thanks, On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: > > > On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: > > Thanks Zhang for your answer, > > I ended up getting a compiling and running TS routine... that does not > give me the answers, and i am not sure what i am doing wrong, > > My TS code so far looks like this: > > (...initialization...) > > call TSCreate(PETSC_COMM_WORLD,ts,ierr) > call TSSetProblemType(ts,TS_NONLINEAR,ierr) > call TSSetSolution(ts,gTemperature,ierr) > call > TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) > > > The second last argument should be the user context. If you pass NULL, the > ctx variable in your FormRHSFunction will be NULL. > > call TSSetType(ts,TSSSP,ierr) > call TSSetTimeStep(ts,dt,ierr) > call TSSetDM(ts,daScalars,ierr) > call TSSetMaxTime(ts,TotalTime,ierr) > call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) > call TSSetFromOptions(ts,ierr) > call TSSetUp(ts,ierr) > > (... then inside main calculations loop i have...) > > call TSSolve(ts,gTemperature,ierr) > > (...and my RHSFunction looks like this...) > > subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) > > use petscvec > use petscts > use petscdmda > use petscdm > > USE utils > USE dmdaobjs > USE probsize > USE modelparams > > implicit none > > TS :: ts > PetscReal :: t > Vec :: ctx,t_loc,rhs_loc > PetscErrorCode :: ierr > > > > call TSGetDM(ts,daScalars,ierr) > > call DMGetLocalVector(daScalars,t_loc,ierr); > > > t_loc is the input, it should be not modified. > > call > DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > call > DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > > > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you want > to scatter ghost points for t_loc, not gTemperature. > > call DMDASolveTExplicit(3) > > call DMGetLocalVector(daScalars,rhs_loc,ierr); > call > DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > call > DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > > > There is no need to scatter ghost points for rhs_loc. > > > > call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) > call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) > > end subroutine FormRHSFunction > > Where DMDASolveTExplicit(3) is the old function to calculate time > integration with runge kutta, modified to only generate the f(t,u) which in > this case is rhs_loc, > > I still have several doubts: > > > - Will this explicit RHS calculation work with TS routines? my time > integration is explicit so far and it would involve a fair deal of extra > work to make it implicit. > > For explicit time integration, one needs to provide only RHSFunction. > > > - This 't' parameter in the RHSFunction is controlled by PETSC? i am > not providing anything for it directly, where is it coming from? > > > It is controlled by PETSc. If your problem is autonomous (the RHS does not > depend on t), it can be simply ignored. > > > > - Do i need to provide a Jacobian or the TS routine will try to guess > one? This is related to the first point where my time scheme being explicit > does not use a jacobian. > > For explicit time integration, Jacobian is not needed. > > Hong > > > Thanks, any help is appreciated, if you see any errors or need more > information please let me know, > > > Regards, > > Manuel > > On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: > >> >> >> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >> Hi PETSc, >> >> I am trying to implement the Time stepping routines in my model, i have a >> working runge-kutta time scheme that goes to the following steps: >> >> >> - Interpolate u,v,w to the time advancing variable position. >> - Calculate nonlinear coefficients and advect velocities with a >> forward-backward shock capturing scheme. >> - Calculate the variable laplacian >> - Sum terms to create RHS (nonlinear advected velocities + laplacian) >> - Finally, the runge kutta integration is done in a typical way that >> looks like: >> >> newtemp(t+1) = prevtemp(t) + dt*RHS >> >> >> So my questions are: >> >> - I think my problem is nonlinear, but is being made linearized by >> the advecting scheme, is this correct? this is to know if i should use the >> linear or nonlinear routines for TS. >> >> TSComputeRHSFunctionLinear is just a convenience function for linear ODEs >> in the form udot=Au. Using it won?t buy you much. So for TS starters, it is >> fine to assume your problem is nonlinear and think of the form udot=f(t,u) >> where f is the RHS function. >> >> >> - How do i know what are the appropriate routines i should be using >> here? so far i think i should use the following: >> >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> call TSSetProblemType(ts,TS_LINEAR,ierr) >> call TSSetTimeStep(ts,dt,ierr) >> >> call TSSetFromOptions(ts,ierr) >> >> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) >> call TSSolve(ts,loctemperature,ierr) >> >> Should i use call TSSetRHSJacobian for the temperature jacobian in this >> case? >> >> >> I would suggest to write your own RHSFunction and set it to TS with >> TSSetRHSFunction(). >> >> >> >> I am using >> https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html as >> a general guide, is there a more appropriate example? >> >> >> ex4 is a good example. In addition, ex9 uses finite volume method with >> slope limiters to solve a variety of problems such as advection equation, >> burgers equation and shallow water equation. It might be an overkill, but >> it seems to be close to the problem you are trying to solve. Note that you >> might want to use the SSP methods (-ts_type ssp) instead of the classic >> Runge-Kutta methods for problems with shocks. >> >> Hong (Mr.) >> >> >> The dt value and total timesteps are controlled by the model, >> >> Thanks for your help, >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jul 8 22:36:33 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jul 2019 03:36:33 +0000 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> Message-ID: <4EB0D523-28C2-4CB0-87CB-1AF8D0FF96BE@anl.gov> > On Jul 8, 2019, at 6:53 PM, Manuel Valera via petsc-users wrote: > > Hi Zhang, > > Thanks to your help i have implemented the TS routine for my temperature DMDA array, and it works correctly including in MPI. > > The breakthrough for me was realizing that TS steps and max time steps are all set up once, and used/advanced all at once when invoked with TSSolution. So what i did was to add all of the TSCreate / initialization into the main loop so i am basically creating and destroying the TS objects every time step. Why? You can create the TS objects once outside and then inside advance them one step at a time inside the loop if you want. > I hope the overhead time is not to bad when i scale to bigger problems. I would say it is only extremely strange conditions would you need to recreate the TS for each timestep. Better to now figure out how to reuse them. > > Next step would be to apply the same to advance the velocities, but problem now is that they are in a 4D dmda array of 3 degrees of freedom, any suggestions on how to implement this? does TS support multiple degrees of freedom arrays? TS doesn't care how you access the vector entries in your function evaluations. All TS every sees is a Vec. I am scared that you said 4D dmda array of 3 degrees of freedom. PETSc DMDA only goes up to 3 dimensions. So if you truely need 4 dim plus 3 dof per point you need to "fake it" but putting the last dimension into the dof slot. In other words use a 3d DMDA and have for dof 3*n where n is the size of the 4th dimension. The code to access the values will now be a little cumbersome since you can only access the array as 4d and the 4th dimension contains both the 3 and the last dimension but it is not too bad. But if you mean 3 physical dimensions plus 3 dof then everything is fine just use the 3d DMDA. Barry > > Thanks, > > On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: > > >> On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: >> >> Thanks Zhang for your answer, >> >> I ended up getting a compiling and running TS routine... that does not give me the answers, and i am not sure what i am doing wrong, >> >> My TS code so far looks like this: >> >> (...initialization...) >> >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) >> call TSSetSolution(ts,gTemperature,ierr) >> call TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) > > The second last argument should be the user context. If you pass NULL, the ctx variable in your FormRHSFunction will be NULL. > >> call TSSetType(ts,TSSSP,ierr) >> call TSSetTimeStep(ts,dt,ierr) >> call TSSetDM(ts,daScalars,ierr) >> call TSSetMaxTime(ts,TotalTime,ierr) >> call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) >> call TSSetFromOptions(ts,ierr) >> call TSSetUp(ts,ierr) >> >> (... then inside main calculations loop i have...) >> >> call TSSolve(ts,gTemperature,ierr) >> >> (...and my RHSFunction looks like this...) >> >> subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) >> >> use petscvec >> use petscts >> use petscdmda >> use petscdm >> >> USE utils >> USE dmdaobjs >> USE probsize >> USE modelparams >> >> implicit none >> >> TS :: ts >> PetscReal :: t >> Vec :: ctx,t_loc,rhs_loc >> PetscErrorCode :: ierr >> >> >> >> call TSGetDM(ts,daScalars,ierr) >> >> call DMGetLocalVector(daScalars,t_loc,ierr); > > t_loc is the input, it should be not modified. > >> call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >> call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you want to scatter ghost points for t_loc, not gTemperature. > >> call DMDASolveTExplicit(3) >> >> call DMGetLocalVector(daScalars,rhs_loc,ierr); >> call DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >> call DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > > There is no need to scatter ghost points for rhs_loc. > >> >> >> call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) >> call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) >> >> end subroutine FormRHSFunction >> >> Where DMDASolveTExplicit(3) is the old function to calculate time integration with runge kutta, modified to only generate the f(t,u) which in this case is rhs_loc, >> >> I still have several doubts: >> >> ? Will this explicit RHS calculation work with TS routines? my time integration is explicit so far and it would involve a fair deal of extra work to make it implicit. > For explicit time integration, one needs to provide only RHSFunction. > >> ? This 't' parameter in the RHSFunction is controlled by PETSC? i am not providing anything for it directly, where is it coming from? > > It is controlled by PETSc. If your problem is autonomous (the RHS does not depend on t), it can be simply ignored. > >> ? Do i need to provide a Jacobian or the TS routine will try to guess one? This is related to the first point where my time scheme being explicit does not use a jacobian. > For explicit time integration, Jacobian is not needed. > > Hong > >> >> Thanks, any help is appreciated, if you see any errors or need more information please let me know, >> >> Regards, >> >> Manuel >> >> On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: >> >> >>> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users wrote: >>> >>> Hi PETSc, >>> >>> I am trying to implement the Time stepping routines in my model, i have a working runge-kutta time scheme that goes to the following steps: >>> >>> ? Interpolate u,v,w to the time advancing variable position. >>> ? Calculate nonlinear coefficients and advect velocities with a forward-backward shock capturing scheme. >>> ? Calculate the variable laplacian >>> ? Sum terms to create RHS (nonlinear advected velocities + laplacian) >>> ? Finally, the runge kutta integration is done in a typical way that looks like: >>> newtemp(t+1) = prevtemp(t) + dt*RHS >>> >>> >>> So my questions are: >>> ? I think my problem is nonlinear, but is being made linearized by the advecting scheme, is this correct? this is to know if i should use the linear or nonlinear routines for TS. >> TSComputeRHSFunctionLinear is just a convenience function for linear ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, it is fine to assume your problem is nonlinear and think of the form udot=f(t,u) where f is the RHS function. >>> ? How do i know what are the appropriate routines i should be using here? so far i think i should use the following: >>> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >>> call TSSetProblemType(ts,TS_LINEAR,ierr) >>> call TSSetTimeStep(ts,dt,ierr) >>> >>> call TSSetFromOptions(ts,ierr) >>> >>> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) >>> call TSSolve(ts,loctemperature,ierr) >>> >>> Should i use call TSSetRHSJacobian for the temperature jacobian in this case? >> >> I would suggest to write your own RHSFunction and set it to TS with TSSetRHSFunction(). >> >>> >>> >>> I am using https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html as a general guide, is there a more appropriate example? >> >> ex4 is a good example. In addition, ex9 uses finite volume method with slope limiters to solve a variety of problems such as advection equation, burgers equation and shallow water equation. It might be an overkill, but it seems to be close to the problem you are trying to solve. Note that you might want to use the SSP methods (-ts_type ssp) instead of the classic Runge-Kutta methods for problems with shocks. >> >> Hong (Mr.) >> >>> >>> The dt value and total timesteps are controlled by the model, >>> >>> Thanks for your help, >>> >>> >> > From dayedut123 at 163.com Tue Jul 9 08:30:44 2019 From: dayedut123 at 163.com (=?GBK?B?ztI=?=) Date: Tue, 9 Jul 2019 21:30:44 +0800 (CST) Subject: [petsc-users] PETSC ERROR: Petsc has generated inconsistent data with PCGAMG and KSPGRMES Message-ID: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> Hello all, I set PCGAMG and KSPGRMES to solve the matrix which is formed by a particle method such as SPH. But when there is always a PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Petsc has generated inconsistent data [2]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 [2]PETSC ERROR: ./MPS on a arch-linux2-cxx-debug named daye-Precision-7920-Tower by daye Tue Jul 9 20:59:39 2019 [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-hypre --with-clanguage=cxx [2]PETSC ERROR: #1 smoothAggs() line 418 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c [2]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 970 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c [2]PETSC ERROR: #3 PCSetUp_GAMG() line 518 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/gamg.c [2]PETSC ERROR: #4 PCSetUp() line 923 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #5 KSPSetUp() line 381 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #6 KSPSolve() line 612 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c ^C[mpiexec at daye-Precision-7920-Tower] Sending Ctrl-C to processes as requested [mpiexec at daye-Precision-7920-Tower] Press Ctrl-C again to force abort This matrix can be solved when I use PCHYPRE and KSPGRMES. But when I change the pre-condition to PCGAMG, this error always occurs. Any suggestions about that? Thank you at first! Daye -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 9 13:27:25 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jul 2019 18:27:25 +0000 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> Message-ID: <929A9A3B-5465-443D-8277-4B44CF5A0636@mcs.anl.gov> > On Jul 8, 2019, at 6:53 PM, Manuel Valera via petsc-users wrote: > > Hi Zhang, > > Thanks to your help i have implemented the TS routine for my temperature DMDA array, and it works correctly including in MPI. > > The breakthrough for me was realizing that TS steps and max time steps are all set up once, and used/advanced all at once when invoked with TSSolution. So what i did was to add all of the TSCreate / initialization into the main loop so i am basically creating and destroying the TS objects every time step. Why? You can create the TS objects once outside and then inside advance them one step at a time inside the loop if you want. > I hope the overhead time is not to bad when i scale to bigger problems. I would say it is only extremely strange conditions would you need to recreate the TS for each timestep. Better to now figure out how to reuse them. > > Next step would be to apply the same to advance the velocities, but problem now is that they are in a 4D dmda array of 3 degrees of freedom, any suggestions on how to implement this? does TS support multiple degrees of freedom arrays? TS doesn't care how you access the vector entries in your function evaluations. All TS every sees is a Vec. I am scared that you said 4D dmda array of 3 degrees of freedom. PETSc DMDA only goes up to 3 dimensions. So if you truely need 4 dim plus 3 dof per point you need to "fake it" but putting the last dimension into the dof slot. In other words use a 3d DMDA and have for dof 3*n where n is the size of the 4th dimension. The code to access the values will now be a little cumbersome since you can only access the array as 4d and the 4th dimension contains both the 3 and the last dimension but it is not too bad. But if you mean 3 physical dimensions plus 3 dof then everything is fine just use the 3d DMDA. Barry > > Thanks, > > On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: > > >> On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: >> >> Thanks Zhang for your answer, >> >> I ended up getting a compiling and running TS routine... that does not give me the answers, and i am not sure what i am doing wrong, >> >> My TS code so far looks like this: >> >> (...initialization...) >> >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) >> call TSSetSolution(ts,gTemperature,ierr) >> call TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) > > The second last argument should be the user context. If you pass NULL, the ctx variable in your FormRHSFunction will be NULL. > >> call TSSetType(ts,TSSSP,ierr) >> call TSSetTimeStep(ts,dt,ierr) >> call TSSetDM(ts,daScalars,ierr) >> call TSSetMaxTime(ts,TotalTime,ierr) >> call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) >> call TSSetFromOptions(ts,ierr) >> call TSSetUp(ts,ierr) >> >> (... then inside main calculations loop i have...) >> >> call TSSolve(ts,gTemperature,ierr) >> >> (...and my RHSFunction looks like this...) >> >> subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) >> >> use petscvec >> use petscts >> use petscdmda >> use petscdm >> >> USE utils >> USE dmdaobjs >> USE probsize >> USE modelparams >> >> implicit none >> >> TS :: ts >> PetscReal :: t >> Vec :: ctx,t_loc,rhs_loc >> PetscErrorCode :: ierr >> >> >> >> call TSGetDM(ts,daScalars,ierr) >> >> call DMGetLocalVector(daScalars,t_loc,ierr); > > t_loc is the input, it should be not modified. > >> call DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >> call DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you want to scatter ghost points for t_loc, not gTemperature. > >> call DMDASolveTExplicit(3) >> >> call DMGetLocalVector(daScalars,rhs_loc,ierr); >> call DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >> call DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > > There is no need to scatter ghost points for rhs_loc. > >> >> >> call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) >> call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) >> >> end subroutine FormRHSFunction >> >> Where DMDASolveTExplicit(3) is the old function to calculate time integration with runge kutta, modified to only generate the f(t,u) which in this case is rhs_loc, >> >> I still have several doubts: >> >> ? Will this explicit RHS calculation work with TS routines? my time integration is explicit so far and it would involve a fair deal of extra work to make it implicit. > For explicit time integration, one needs to provide only RHSFunction. > >> ? This 't' parameter in the RHSFunction is controlled by PETSC? i am not providing anything for it directly, where is it coming from? > > It is controlled by PETSc. If your problem is autonomous (the RHS does not depend on t), it can be simply ignored. > >> ? Do i need to provide a Jacobian or the TS routine will try to guess one? This is related to the first point where my time scheme being explicit does not use a jacobian. > For explicit time integration, Jacobian is not needed. > > Hong > >> >> Thanks, any help is appreciated, if you see any errors or need more information please let me know, >> >> Regards, >> >> Manuel >> >> On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: >> >> >>> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users wrote: >>> >>> Hi PETSc, >>> >>> I am trying to implement the Time stepping routines in my model, i have a working runge-kutta time scheme that goes to the following steps: >>> >>> ? Interpolate u,v,w to the time advancing variable position. >>> ? Calculate nonlinear coefficients and advect velocities with a forward-backward shock capturing scheme. >>> ? Calculate the variable laplacian >>> ? Sum terms to create RHS (nonlinear advected velocities + laplacian) >>> ? Finally, the runge kutta integration is done in a typical way that looks like: >>> newtemp(t+1) = prevtemp(t) + dt*RHS >>> >>> >>> So my questions are: >>> ? I think my problem is nonlinear, but is being made linearized by the advecting scheme, is this correct? this is to know if i should use the linear or nonlinear routines for TS. >> TSComputeRHSFunctionLinear is just a convenience function for linear ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, it is fine to assume your problem is nonlinear and think of the form udot=f(t,u) where f is the RHS function. >>> ? How do i know what are the appropriate routines i should be using here? so far i think i should use the following: >>> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >>> call TSSetProblemType(ts,TS_LINEAR,ierr) >>> call TSSetTimeStep(ts,dt,ierr) >>> >>> call TSSetFromOptions(ts,ierr) >>> >>> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) >>> call TSSolve(ts,loctemperature,ierr) >>> >>> Should i use call TSSetRHSJacobian for the temperature jacobian in this case? >> >> I would suggest to write your own RHSFunction and set it to TS with TSSetRHSFunction(). >> >>> >>> >>> I am using https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html as a general guide, is there a more appropriate example? >> >> ex4 is a good example. In addition, ex9 uses finite volume method with slope limiters to solve a variety of problems such as advection equation, burgers equation and shallow water equation. It might be an overkill, but it seems to be close to the problem you are trying to solve. Note that you might want to use the SSP methods (-ts_type ssp) instead of the classic Runge-Kutta methods for problems with shocks. >> >> Hong (Mr.) >> >>> >>> The dt value and total timesteps are controlled by the model, >>> >>> Thanks for your help, >>> >>> >> > From bsmith at mcs.anl.gov Tue Jul 9 14:36:55 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jul 2019 19:36:55 +0000 Subject: [petsc-users] PETSC ERROR: Petsc has generated inconsistent data with PCGAMG and KSPGRMES In-Reply-To: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> References: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> Message-ID: <4DDB22E9-1A8E-45A7-9919-C9063D8C38A0@anl.gov> Most preconditioners don't care if the matrix has non-symmetric non-zero structure. GAMG requires it have a symmetric nonzero structure. It lists some options you can try to make it symmetric. > [2]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. > On Jul 9, 2019, at 8:30 AM, ? via petsc-users wrote: > > Hello all, > I set PCGAMG and KSPGRMES to solve the matrix which is formed by a particle method such as SPH. But when there is always a PETSC ERROR: > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [2]PETSC ERROR: Petsc has generated inconsistent data > [2]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 > [2]PETSC ERROR: ./MPS on a arch-linux2-cxx-debug named daye-Precision-7920-Tower by daye Tue Jul 9 20:59:39 2019 > [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-hypre --with-clanguage=cxx > [2]PETSC ERROR: #1 smoothAggs() line 418 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c > [2]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 970 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c > [2]PETSC ERROR: #3 PCSetUp_GAMG() line 518 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/gamg.c > [2]PETSC ERROR: #4 PCSetUp() line 923 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: #5 KSPSetUp() line 381 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: #6 KSPSolve() line 612 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c > ^C[mpiexec at daye-Precision-7920-Tower] Sending Ctrl-C to processes as requested > [mpiexec at daye-Precision-7920-Tower] Press Ctrl-C again to force abort > > This matrix can be solved when I use PCHYPRE and KSPGRMES. But when I change the pre-condition to PCGAMG, this error always occurs. Any suggestions about that? > Thank you at first! > > Daye > > > From mvalera-w at sdsu.edu Tue Jul 9 14:37:25 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Tue, 9 Jul 2019 12:37:25 -0700 Subject: [petsc-users] Implementing TS routine In-Reply-To: <929A9A3B-5465-443D-8277-4B44CF5A0636@mcs.anl.gov> References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> <929A9A3B-5465-443D-8277-4B44CF5A0636@mcs.anl.gov> Message-ID: On Tue, Jul 9, 2019 at 11:27 AM Smith, Barry F. wrote: > > > > On Jul 8, 2019, at 6:53 PM, Manuel Valera via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hi Zhang, > > > > Thanks to your help i have implemented the TS routine for my temperature > DMDA array, and it works correctly including in MPI. > > > > The breakthrough for me was realizing that TS steps and max time steps > are all set up once, and used/advanced all at once when invoked with > TSSolution. So what i did was to add all of the TSCreate / initialization > into the main loop so i am basically creating and destroying the TS objects > every time step. > > Why? You can create the TS objects once outside and then inside advance > them one step at a time inside the loop if you want. > For some reason this is not possible, if you could provide any ideas let me know, the situation is this: This block is now inside the main loop: call TSCreate(PETSC_COMM_WORLD,ts,ierr) call TSSetProblemType(ts,TS_NONLINEAR,ierr) call TSSetSolution(ts,LocTemperature,ierr) call TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) call TSSetType(ts,TSSSP,ierr) call TSSetTimeStep(ts,dt,ierr) call TSSetDM(ts,daScalars,ierr) call TSSetMaxTime(ts,dt,ierr) call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_MATCHSTEP,ierr) call TSSetFromOptions(ts,ierr) call TSSetUp(ts,ierr) call TSSolve(ts,LocTemperature,ierr) call TSDestroy(ts,ierr) Where i create, set everything up and solve the TS problem for a single time step. If i take TSCreate and TSDestroy out of this loop, the TS will advance all the dt's at time zero of the program (first iteration) and then continue executing the rest without taking this into account. The only way i have found for TS to be executed every time in the way i need it is creating all objects and executing them at this moment. Any idea on how i can control this from out of the loop would be appreciated. Keep in mind that i want to use only the TS routine in this step, as i mentioned every example seems to include the dynamics of the problem inside the TS, and then it makes sense that we could set a specific totaltime and advance it all at TSSolution, what i am trying to do is quite different, as all of the dynamics are to be preserved outside of the TS, i only really need to make use of the function that takes u and f(t,u) and gives out u(t+1) with a specific time stepping scheme. > > > I hope the overhead time is not to bad when i scale to bigger problems. > > I would say it is only extremely strange conditions would you need to > recreate the TS for each timestep. Better to now figure out how to reuse > them. > Agreed and i would like to know how to reuse them but the behavior seems to be against what i want to do. > > > Next step would be to apply the same to advance the velocities, but > problem now is that they are in a 4D dmda array of 3 degrees of freedom, > any suggestions on how to implement this? does TS support multiple degrees > of freedom arrays? > > TS doesn't care how you access the vector entries in your function > evaluations. All TS every sees is a Vec. > > I am scared that you said 4D dmda array of 3 degrees of freedom. PETSc > DMDA only goes up to 3 dimensions. So if you truely need 4 dim plus 3 dof > per point you need to "fake it" but putting the last dimension into the dof > slot. In other words use a 3d DMDA and have for dof 3*n where n is the > size of the 4th dimension. The code to access the values will now be a > little cumbersome since you can only access the array as 4d and the 4th > dimension contains both the 3 and the last dimension but it is not too bad. > > But if you mean 3 physical dimensions plus 3 dof then everything is > fine just use the 3d DMDA. > Sorry i didn't speak clearly, i really meant 3 physical dimensions plus 3 DOF, i am already using the 3D DMDAs here. My velocity array looks like vel(0:3,i,j,k) so 3 spatial dimensions and 3 degrees of freedom one for each velocity direction, so vel(0,...) is u, vel(1,..) is v and vel(2,..) is w. The problem comes again when i try to use the TS to solve for this kind of array. Do i need a 1 dimensional vector for the f(t,u) ? In the case of the temperature, i have created a natural vector from the 3D RHS array, and use that as a f(t,u), do i need to do the same for the velocities, or can i provide a 3D array with 3 DOF as either the f(t,u) and the u? If the latter is the case, i could create a new 3D DMDA to hold the RHS functions of each of the velocity, in the same structure the velocities are. If the case if the former i would need to create new natural vectors for each of the RHS, but also copy the velocities into a single DOF DMDA ? Thanks, i think i am a bit at a loss here, Regards, > > Barry > > > > > > > Thanks, > > > > On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: > > > > > >> On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: > >> > >> Thanks Zhang for your answer, > >> > >> I ended up getting a compiling and running TS routine... that does not > give me the answers, and i am not sure what i am doing wrong, > >> > >> My TS code so far looks like this: > >> > >> (...initialization...) > >> > >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) > >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) > >> call TSSetSolution(ts,gTemperature,ierr) > >> call > TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) > > > > The second last argument should be the user context. If you pass NULL, > the ctx variable in your FormRHSFunction will be NULL. > > > >> call TSSetType(ts,TSSSP,ierr) > >> call TSSetTimeStep(ts,dt,ierr) > >> call TSSetDM(ts,daScalars,ierr) > >> call TSSetMaxTime(ts,TotalTime,ierr) > >> call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) > >> call TSSetFromOptions(ts,ierr) > >> call TSSetUp(ts,ierr) > >> > >> (... then inside main calculations loop i have...) > >> > >> call TSSolve(ts,gTemperature,ierr) > >> > >> (...and my RHSFunction looks like this...) > >> > >> subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) > >> > >> use petscvec > >> use petscts > >> use petscdmda > >> use petscdm > >> > >> USE utils > >> USE dmdaobjs > >> USE probsize > >> USE modelparams > >> > >> implicit none > >> > >> TS :: ts > >> PetscReal :: t > >> Vec :: ctx,t_loc,rhs_loc > >> PetscErrorCode :: ierr > >> > >> > >> > >> call TSGetDM(ts,daScalars,ierr) > >> > >> call DMGetLocalVector(daScalars,t_loc,ierr); > > > > t_loc is the input, it should be not modified. > > > >> call > DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > >> call > DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) > > > > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you > want to scatter ghost points for t_loc, not gTemperature. > > > >> call DMDASolveTExplicit(3) > >> > >> call DMGetLocalVector(daScalars,rhs_loc,ierr); > >> call > DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > >> call > DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) > > > > There is no need to scatter ghost points for rhs_loc. > > > >> > >> > >> call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) > >> call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) > >> > >> end subroutine FormRHSFunction > >> > >> Where DMDASolveTExplicit(3) is the old function to calculate time > integration with runge kutta, modified to only generate the f(t,u) which in > this case is rhs_loc, > >> > >> I still have several doubts: > >> > >> ? Will this explicit RHS calculation work with TS routines? my > time integration is explicit so far and it would involve a fair deal of > extra work to make it implicit. > > For explicit time integration, one needs to provide only RHSFunction. > > > >> ? This 't' parameter in the RHSFunction is controlled by PETSC? i > am not providing anything for it directly, where is it coming from? > > > > It is controlled by PETSc. If your problem is autonomous (the RHS does > not depend on t), it can be simply ignored. > > > >> ? Do i need to provide a Jacobian or the TS routine will try to > guess one? This is related to the first point where my time scheme being > explicit does not use a jacobian. > > For explicit time integration, Jacobian is not needed. > > > > Hong > > > >> > >> Thanks, any help is appreciated, if you see any errors or need more > information please let me know, > >> > >> Regards, > >> > >> Manuel > >> > >> On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: > >> > >> > >>> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >>> > >>> Hi PETSc, > >>> > >>> I am trying to implement the Time stepping routines in my model, i > have a working runge-kutta time scheme that goes to the following steps: > >>> > >>> ? Interpolate u,v,w to the time advancing variable position. > >>> ? Calculate nonlinear coefficients and advect velocities with a > forward-backward shock capturing scheme. > >>> ? Calculate the variable laplacian > >>> ? Sum terms to create RHS (nonlinear advected velocities + > laplacian) > >>> ? Finally, the runge kutta integration is done in a typical way > that looks like: > >>> newtemp(t+1) = prevtemp(t) + dt*RHS > >>> > >>> > >>> So my questions are: > >>> ? I think my problem is nonlinear, but is being made linearized by > the advecting scheme, is this correct? this is to know if i should use the > linear or nonlinear routines for TS. > >> TSComputeRHSFunctionLinear is just a convenience function for linear > ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, > it is fine to assume your problem is nonlinear and think of the form > udot=f(t,u) where f is the RHS function. > >>> ? How do i know what are the appropriate routines i should be > using here? so far i think i should use the following: > >>> call TSCreate(PETSC_COMM_WORLD,ts,ierr) > >>> call TSSetProblemType(ts,TS_LINEAR,ierr) > >>> call TSSetTimeStep(ts,dt,ierr) > >>> > >>> call TSSetFromOptions(ts,ierr) > >>> > >>> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) > >>> call TSSolve(ts,loctemperature,ierr) > >>> > >>> Should i use call TSSetRHSJacobian for the temperature jacobian in > this case? > >> > >> I would suggest to write your own RHSFunction and set it to TS with > TSSetRHSFunction(). > >> > >>> > >>> > >>> I am using > https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html > as a general guide, is there a more appropriate example? > >> > >> ex4 is a good example. In addition, ex9 uses finite volume method with > slope limiters to solve a variety of problems such as advection equation, > burgers equation and shallow water equation. It might be an overkill, but > it seems to be close to the problem you are trying to solve. Note that you > might want to use the SSP methods (-ts_type ssp) instead of the classic > Runge-Kutta methods for problems with shocks. > >> > >> Hong (Mr.) > >> > >>> > >>> The dt value and total timesteps are controlled by the model, > >>> > >>> Thanks for your help, > >>> > >>> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 9 15:31:59 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Jul 2019 15:31:59 -0500 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> <929A9A3B-5465-443D-8277-4B44CF5A0636@mcs.anl.gov> Message-ID: On Tue, Jul 9, 2019 at 2:39 PM Manuel Valera via petsc-users < petsc-users at mcs.anl.gov> wrote: > On Tue, Jul 9, 2019 at 11:27 AM Smith, Barry F. > wrote: > >> > On Jul 8, 2019, at 6:53 PM, Manuel Valera via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > >> > Hi Zhang, >> > >> > Thanks to your help i have implemented the TS routine for my >> temperature DMDA array, and it works correctly including in MPI. >> > >> > The breakthrough for me was realizing that TS steps and max time steps >> are all set up once, and used/advanced all at once when invoked with >> TSSolution. So what i did was to add all of the TSCreate / initialization >> into the main loop so i am basically creating and destroying the TS objects >> every time step. >> >> Why? You can create the TS objects once outside and then inside >> advance them one step at a time inside the loop if you want. >> > > For some reason this is not possible, if you could provide any ideas let > me know, the situation is this: > > This block is now inside the main loop: > call TSCreate(PETSC_COMM_WORLD,ts,ierr) > call TSSetProblemType(ts,TS_NONLINEAR,ierr) > call TSSetSolution(ts,LocTemperature,ierr) > call > TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) > call TSSetType(ts,TSSSP,ierr) > call TSSetTimeStep(ts,dt,ierr) > call TSSetDM(ts,daScalars,ierr) > call TSSetMaxTime(ts,dt,ierr) > call > TSSetExactFinalTime(ts,TS_EXACTFINALTIME_MATCHSTEP,ierr) > call TSSetFromOptions(ts,ierr) > call TSSetUp(ts,ierr) > call TSSolve(ts,LocTemperature,ierr) > call TSDestroy(ts,ierr) > > Where i create, set everything up and solve the TS problem for a single > time step. > I admit that I have not completely followed the discussion. I will try and ask a few questions so that I can understand. First, you can use TSStep(), instead of TSSolve(), in order to run a single time step. > If i take TSCreate and TSDestroy out of this loop, the TS will advance all > the dt's at time zero of the program (first iteration) and then continue > executing the rest without taking this into account. The only way i have > found for TS to be executed every time in the way i need > Is it easy to say what you need to do between steps? > it is creating all objects and executing them at this moment. Any idea on > how i can control this from out of the loop would be appreciated. > > Keep in mind that i want to use only the TS routine in this step, as i > mentioned every example seems to include the dynamics of the problem inside > the TS, and then it makes sense that we could set a specific totaltime and > advance it all at TSSolution, what i am trying to do is quite different, as > all of the dynamics are to be preserved outside of the TS, i only really > need to make use of the function that takes u and f(t,u) and gives out > u(t+1) with a specific time stepping scheme. > >> >> > I hope the overhead time is not to bad when i scale to bigger problems. >> >> I would say it is only extremely strange conditions would you need to >> recreate the TS for each timestep. Better to now figure out how to reuse >> them. >> > > Agreed and i would like to know how to reuse them but the behavior seems > to be against what i want to do. > > > > >> > Next step would be to apply the same to advance the velocities, but >> problem now is that they are in a 4D dmda array of 3 degrees of freedom, >> any suggestions on how to implement this? does TS support multiple degrees >> of freedom arrays? >> >> TS doesn't care how you access the vector entries in your function >> evaluations. All TS every sees is a Vec. >> >> I am scared that you said 4D dmda array of 3 degrees of freedom. PETSc >> DMDA only goes up to 3 dimensions. So if you truely need 4 dim plus 3 dof >> per point you need to "fake it" but putting the last dimension into the dof >> slot. In other words use a 3d DMDA and have for dof 3*n where n is the >> size of the 4th dimension. The code to access the values will now be a >> little cumbersome since you can only access the array as 4d and the 4th >> dimension contains both the 3 and the last dimension but it is not too bad. >> >> But if you mean 3 physical dimensions plus 3 dof then everything is >> fine just use the 3d DMDA. >> > > Sorry i didn't speak clearly, i really meant 3 physical dimensions plus 3 > DOF, i am already using the 3D DMDAs here. My velocity array looks like > vel(0:3,i,j,k) so 3 spatial dimensions and 3 degrees of freedom one for > each velocity direction, so vel(0,...) is u, vel(1,..) is v and vel(2,..) > is w. The problem comes again when i try to use the TS to solve for this > kind of array. Do i need a 1 dimensional vector for the f(t,u) ? > There should be no problem with multiple DOF per point in your domain. You just take this into account in your residual (and Jacobian) function. There are many examples in TS with multiple DOFs. Maybe there is something I am missing here. Thanks, Matt > In the case of the temperature, i have created a natural vector from the > 3D RHS array, and use that as a f(t,u), do i need to do the same for the > velocities, or can i provide a 3D array with 3 DOF as either the f(t,u) and > the u? If the latter is the case, i could create a new 3D DMDA to hold the > RHS functions of each of the velocity, in the same structure the velocities > are. If the case if the former i would need to create new natural vectors > for each of the RHS, but also copy the velocities into a single DOF DMDA ? > > Thanks, i think i am a bit at a loss here, > > Regards, > > > > > > > >> >> Barry >> >> >> >> > >> > Thanks, >> > >> > On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: >> > >> > >> >> On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: >> >> >> >> Thanks Zhang for your answer, >> >> >> >> I ended up getting a compiling and running TS routine... that does not >> give me the answers, and i am not sure what i am doing wrong, >> >> >> >> My TS code so far looks like this: >> >> >> >> (...initialization...) >> >> >> >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) >> >> call TSSetSolution(ts,gTemperature,ierr) >> >> call >> TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) >> > >> > The second last argument should be the user context. If you pass NULL, >> the ctx variable in your FormRHSFunction will be NULL. >> > >> >> call TSSetType(ts,TSSSP,ierr) >> >> call TSSetTimeStep(ts,dt,ierr) >> >> call TSSetDM(ts,daScalars,ierr) >> >> call TSSetMaxTime(ts,TotalTime,ierr) >> >> call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) >> >> call TSSetFromOptions(ts,ierr) >> >> call TSSetUp(ts,ierr) >> >> >> >> (... then inside main calculations loop i have...) >> >> >> >> call TSSolve(ts,gTemperature,ierr) >> >> >> >> (...and my RHSFunction looks like this...) >> >> >> >> subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) >> >> >> >> use petscvec >> >> use petscts >> >> use petscdmda >> >> use petscdm >> >> >> >> USE utils >> >> USE dmdaobjs >> >> USE probsize >> >> USE modelparams >> >> >> >> implicit none >> >> >> >> TS :: ts >> >> PetscReal :: t >> >> Vec :: ctx,t_loc,rhs_loc >> >> PetscErrorCode :: ierr >> >> >> >> >> >> >> >> call TSGetDM(ts,daScalars,ierr) >> >> >> >> call DMGetLocalVector(daScalars,t_loc,ierr); >> > >> > t_loc is the input, it should be not modified. >> > >> >> call >> DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >> >> call >> DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >> > >> > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you >> want to scatter ghost points for t_loc, not gTemperature. >> > >> >> call DMDASolveTExplicit(3) >> >> >> >> call DMGetLocalVector(daScalars,rhs_loc,ierr); >> >> call >> DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >> >> call >> DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >> > >> > There is no need to scatter ghost points for rhs_loc. >> > >> >> >> >> >> >> call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) >> >> call DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) >> >> >> >> end subroutine FormRHSFunction >> >> >> >> Where DMDASolveTExplicit(3) is the old function to calculate time >> integration with runge kutta, modified to only generate the f(t,u) which in >> this case is rhs_loc, >> >> >> >> I still have several doubts: >> >> >> >> ? Will this explicit RHS calculation work with TS routines? my >> time integration is explicit so far and it would involve a fair deal of >> extra work to make it implicit. >> > For explicit time integration, one needs to provide only RHSFunction. >> > >> >> ? This 't' parameter in the RHSFunction is controlled by PETSC? i >> am not providing anything for it directly, where is it coming from? >> > >> > It is controlled by PETSc. If your problem is autonomous (the RHS does >> not depend on t), it can be simply ignored. >> > >> >> ? Do i need to provide a Jacobian or the TS routine will try to >> guess one? This is related to the first point where my time scheme being >> explicit does not use a jacobian. >> > For explicit time integration, Jacobian is not needed. >> > >> > Hong >> > >> >> >> >> Thanks, any help is appreciated, if you see any errors or need more >> information please let me know, >> >> >> >> Regards, >> >> >> >> Manuel >> >> >> >> On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong wrote: >> >> >> >> >> >>> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> >> >>> Hi PETSc, >> >>> >> >>> I am trying to implement the Time stepping routines in my model, i >> have a working runge-kutta time scheme that goes to the following steps: >> >>> >> >>> ? Interpolate u,v,w to the time advancing variable position. >> >>> ? Calculate nonlinear coefficients and advect velocities with a >> forward-backward shock capturing scheme. >> >>> ? Calculate the variable laplacian >> >>> ? Sum terms to create RHS (nonlinear advected velocities + >> laplacian) >> >>> ? Finally, the runge kutta integration is done in a typical way >> that looks like: >> >>> newtemp(t+1) = prevtemp(t) + dt*RHS >> >>> >> >>> >> >>> So my questions are: >> >>> ? I think my problem is nonlinear, but is being made linearized >> by the advecting scheme, is this correct? this is to know if i should use >> the linear or nonlinear routines for TS. >> >> TSComputeRHSFunctionLinear is just a convenience function for linear >> ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, >> it is fine to assume your problem is nonlinear and think of the form >> udot=f(t,u) where f is the RHS function. >> >>> ? How do i know what are the appropriate routines i should be >> using here? so far i think i should use the following: >> >>> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> >>> call TSSetProblemType(ts,TS_LINEAR,ierr) >> >>> call TSSetTimeStep(ts,dt,ierr) >> >>> >> >>> call TSSetFromOptions(ts,ierr) >> >>> >> >>> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) >> >>> call TSSolve(ts,loctemperature,ierr) >> >>> >> >>> Should i use call TSSetRHSJacobian for the temperature jacobian in >> this case? >> >> >> >> I would suggest to write your own RHSFunction and set it to TS with >> TSSetRHSFunction(). >> >> >> >>> >> >>> >> >>> I am using >> https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html >> as a general guide, is there a more appropriate example? >> >> >> >> ex4 is a good example. In addition, ex9 uses finite volume method with >> slope limiters to solve a variety of problems such as advection equation, >> burgers equation and shallow water equation. It might be an overkill, but >> it seems to be close to the problem you are trying to solve. Note that you >> might want to use the SSP methods (-ts_type ssp) instead of the classic >> Runge-Kutta methods for problems with shocks. >> >> >> >> Hong (Mr.) >> >> >> >>> >> >>> The dt value and total timesteps are controlled by the model, >> >>> >> >>> Thanks for your help, >> >>> >> >>> >> >> >> > >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 9 15:44:40 2019 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Jul 2019 16:44:40 -0400 Subject: [petsc-users] PETSC ERROR: Petsc has generated inconsistent data with PCGAMG and KSPGRMES In-Reply-To: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> References: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> Message-ID: On Tue, Jul 9, 2019 at 12:43 PM ? via petsc-users wrote: > Hello all, > I set PCGAMG and KSPGRMES to solve the matrix which is formed by a > particle method such as SPH. But when there is always a PETSC ERROR: > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Petsc has generated inconsistent data > [2]PETSC ERROR: Have un-symmetric graph (apparently). Use > '-(null)pc_gamg_sym_graph true' to symetrize the graph or > '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. > You need to add this. This will symmetrize the graph. I would guess that hypre does this by default. You can just add this to your .petscrc file to essentially make it the default. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Tue Jul 9 18:30:34 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Tue, 9 Jul 2019 16:30:34 -0700 Subject: [petsc-users] Implementing TS routine In-Reply-To: References: <37E620B7-6856-41F7-ADEA-6D86A7FDEE88@anl.gov> <929A9A3B-5465-443D-8277-4B44CF5A0636@mcs.anl.gov> Message-ID: Ok thanks everyone for your replies, Matt thanks for suggesting TStep, it solved my problem for reusing the TS objects. The TS routine are working as intended, and i was able to implement the TS for the velocities as well, i dropped the use of the natural vector and i am using just the DM global vectors as function and input for the TS RHS function. I'll write back if i have further questions, Thanks so much, On Tue, Jul 9, 2019 at 1:32 PM Matthew Knepley wrote: > On Tue, Jul 9, 2019 at 2:39 PM Manuel Valera via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> On Tue, Jul 9, 2019 at 11:27 AM Smith, Barry F. >> wrote: >> >>> > On Jul 8, 2019, at 6:53 PM, Manuel Valera via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> > >>> > Hi Zhang, >>> > >>> > Thanks to your help i have implemented the TS routine for my >>> temperature DMDA array, and it works correctly including in MPI. >>> > >>> > The breakthrough for me was realizing that TS steps and max time steps >>> are all set up once, and used/advanced all at once when invoked with >>> TSSolution. So what i did was to add all of the TSCreate / initialization >>> into the main loop so i am basically creating and destroying the TS objects >>> every time step. >>> >>> Why? You can create the TS objects once outside and then inside >>> advance them one step at a time inside the loop if you want. >>> >> >> For some reason this is not possible, if you could provide any ideas let >> me know, the situation is this: >> >> This block is now inside the main loop: >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) >> call TSSetSolution(ts,LocTemperature,ierr) >> call >> TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) >> call TSSetType(ts,TSSSP,ierr) >> call TSSetTimeStep(ts,dt,ierr) >> call TSSetDM(ts,daScalars,ierr) >> call TSSetMaxTime(ts,dt,ierr) >> call >> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_MATCHSTEP,ierr) >> call TSSetFromOptions(ts,ierr) >> call TSSetUp(ts,ierr) >> call TSSolve(ts,LocTemperature,ierr) >> call TSDestroy(ts,ierr) >> >> Where i create, set everything up and solve the TS problem for a single >> time step. >> > > I admit that I have not completely followed the discussion. I will try and > ask a few questions so that I can understand. > > First, you can use TSStep(), instead of TSSolve(), in order to run a > single time step. > > >> If i take TSCreate and TSDestroy out of this loop, the TS will advance >> all the dt's at time zero of the program (first iteration) and then >> continue executing the rest without taking this into account. The only way >> i have found for TS to be executed every time in the way i need >> > > Is it easy to say what you need to do between steps? > > >> it is creating all objects and executing them at this moment. Any idea on >> how i can control this from out of the loop would be appreciated. >> >> Keep in mind that i want to use only the TS routine in this step, as i >> mentioned every example seems to include the dynamics of the problem inside >> the TS, and then it makes sense that we could set a specific totaltime and >> advance it all at TSSolution, what i am trying to do is quite different, as >> all of the dynamics are to be preserved outside of the TS, i only really >> need to make use of the function that takes u and f(t,u) and gives out >> u(t+1) with a specific time stepping scheme. >> >>> >>> > I hope the overhead time is not to bad when i scale to bigger problems. >>> >>> I would say it is only extremely strange conditions would you need to >>> recreate the TS for each timestep. Better to now figure out how to reuse >>> them. >>> >> >> Agreed and i would like to know how to reuse them but the behavior seems >> to be against what i want to do. >> >> >> > >>> > Next step would be to apply the same to advance the velocities, but >>> problem now is that they are in a 4D dmda array of 3 degrees of freedom, >>> any suggestions on how to implement this? does TS support multiple degrees >>> of freedom arrays? >>> >>> TS doesn't care how you access the vector entries in your function >>> evaluations. All TS every sees is a Vec. >>> >>> I am scared that you said 4D dmda array of 3 degrees of freedom. >>> PETSc DMDA only goes up to 3 dimensions. So if you truely need 4 dim plus 3 >>> dof per point you need to "fake it" but putting the last dimension into the >>> dof slot. In other words use a 3d DMDA and have for dof 3*n where n is the >>> size of the 4th dimension. The code to access the values will now be a >>> little cumbersome since you can only access the array as 4d and the 4th >>> dimension contains both the 3 and the last dimension but it is not too bad. >>> >>> But if you mean 3 physical dimensions plus 3 dof then everything is >>> fine just use the 3d DMDA. >>> >> >> Sorry i didn't speak clearly, i really meant 3 physical dimensions plus 3 >> DOF, i am already using the 3D DMDAs here. My velocity array looks like >> vel(0:3,i,j,k) so 3 spatial dimensions and 3 degrees of freedom one for >> each velocity direction, so vel(0,...) is u, vel(1,..) is v and vel(2,..) >> is w. The problem comes again when i try to use the TS to solve for this >> kind of array. Do i need a 1 dimensional vector for the f(t,u) ? >> > > There should be no problem with multiple DOF per point in your domain. You > just take this into account > in your residual (and Jacobian) function. There are many examples in TS > with multiple DOFs. Maybe there > is something I am missing here. > > Thanks, > > Matt > > >> In the case of the temperature, i have created a natural vector from the >> 3D RHS array, and use that as a f(t,u), do i need to do the same for the >> velocities, or can i provide a 3D array with 3 DOF as either the f(t,u) and >> the u? If the latter is the case, i could create a new 3D DMDA to hold the >> RHS functions of each of the velocity, in the same structure the velocities >> are. If the case if the former i would need to create new natural vectors >> for each of the RHS, but also copy the velocities into a single DOF DMDA ? >> >> Thanks, i think i am a bit at a loss here, >> >> Regards, >> >> >> >> >> >> >> >>> >>> Barry >>> >>> >>> >>> > >>> > Thanks, >>> > >>> > On Thu, Jul 4, 2019 at 9:18 PM Zhang, Hong wrote: >>> > >>> > >>> >> On Jul 3, 2019, at 3:10 PM, Manuel Valera wrote: >>> >> >>> >> Thanks Zhang for your answer, >>> >> >>> >> I ended up getting a compiling and running TS routine... that does >>> not give me the answers, and i am not sure what i am doing wrong, >>> >> >>> >> My TS code so far looks like this: >>> >> >>> >> (...initialization...) >>> >> >>> >> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >>> >> call TSSetProblemType(ts,TS_NONLINEAR,ierr) >>> >> call TSSetSolution(ts,gTemperature,ierr) >>> >> call >>> TSSetRHSFunction(ts,PETSC_NULL_VEC,FormRHSFunction,PETSC_NULL_VEC,ierr) >>> > >>> > The second last argument should be the user context. If you pass NULL, >>> the ctx variable in your FormRHSFunction will be NULL. >>> > >>> >> call TSSetType(ts,TSSSP,ierr) >>> >> call TSSetTimeStep(ts,dt,ierr) >>> >> call TSSetDM(ts,daScalars,ierr) >>> >> call TSSetMaxTime(ts,TotalTime,ierr) >>> >> call TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER,ierr) >>> >> call TSSetFromOptions(ts,ierr) >>> >> call TSSetUp(ts,ierr) >>> >> >>> >> (... then inside main calculations loop i have...) >>> >> >>> >> call TSSolve(ts,gTemperature,ierr) >>> >> >>> >> (...and my RHSFunction looks like this...) >>> >> >>> >> subroutine FormRHSFunction(ts,t,t_loc,rhs_loc,ctx,ierr) >>> >> >>> >> use petscvec >>> >> use petscts >>> >> use petscdmda >>> >> use petscdm >>> >> >>> >> USE utils >>> >> USE dmdaobjs >>> >> USE probsize >>> >> USE modelparams >>> >> >>> >> implicit none >>> >> >>> >> TS :: ts >>> >> PetscReal :: t >>> >> Vec :: ctx,t_loc,rhs_loc >>> >> PetscErrorCode :: ierr >>> >> >>> >> >>> >> >>> >> call TSGetDM(ts,daScalars,ierr) >>> >> >>> >> call DMGetLocalVector(daScalars,t_loc,ierr); >>> > >>> > t_loc is the input, it should be not modified. >>> > >>> >> call >>> DMGlobalToLocalBegin(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >>> >> call >>> DMGlobalToLocalEnd(daScalars,gTemperature,INSERT_VALUES,t_loc,ierr);CHKERRQ(ierr) >>> > >>> > FormRHSFunction is supposed to implement rhs_loc = f(t,t_loc). So you >>> want to scatter ghost points for t_loc, not gTemperature. >>> > >>> >> call DMDASolveTExplicit(3) >>> >> >>> >> call DMGetLocalVector(daScalars,rhs_loc,ierr); >>> >> call >>> DMGlobalToLocalBegin(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >>> >> call >>> DMGlobalToLocalEnd(daScalars,TRHSS_ts,INSERT_VALUES,rhs_loc,ierr);CHKERRQ(ierr) >>> > >>> > There is no need to scatter ghost points for rhs_loc. >>> > >>> >> >>> >> >>> >> call DMRestoreLocalVector(daScalars,t_loc,ierr);CHKERRQ(ierr) >>> >> call >>> DMRestoreLocalVector(daScalars,rhs_loc,ierr);CHKERRQ(ierr) >>> >> >>> >> end subroutine FormRHSFunction >>> >> >>> >> Where DMDASolveTExplicit(3) is the old function to calculate time >>> integration with runge kutta, modified to only generate the f(t,u) which in >>> this case is rhs_loc, >>> >> >>> >> I still have several doubts: >>> >> >>> >> ? Will this explicit RHS calculation work with TS routines? my >>> time integration is explicit so far and it would involve a fair deal of >>> extra work to make it implicit. >>> > For explicit time integration, one needs to provide only RHSFunction. >>> > >>> >> ? This 't' parameter in the RHSFunction is controlled by PETSC? >>> i am not providing anything for it directly, where is it coming from? >>> > >>> > It is controlled by PETSc. If your problem is autonomous (the RHS does >>> not depend on t), it can be simply ignored. >>> > >>> >> ? Do i need to provide a Jacobian or the TS routine will try to >>> guess one? This is related to the first point where my time scheme being >>> explicit does not use a jacobian. >>> > For explicit time integration, Jacobian is not needed. >>> > >>> > Hong >>> > >>> >> >>> >> Thanks, any help is appreciated, if you see any errors or need more >>> information please let me know, >>> >> >>> >> Regards, >>> >> >>> >> Manuel >>> >> >>> >> On Wed, Jun 26, 2019 at 9:54 PM Zhang, Hong >>> wrote: >>> >> >>> >> >>> >>> On Jun 26, 2019, at 4:17 PM, Manuel Valera via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>> >>> >>> Hi PETSc, >>> >>> >>> >>> I am trying to implement the Time stepping routines in my model, i >>> have a working runge-kutta time scheme that goes to the following steps: >>> >>> >>> >>> ? Interpolate u,v,w to the time advancing variable position. >>> >>> ? Calculate nonlinear coefficients and advect velocities with a >>> forward-backward shock capturing scheme. >>> >>> ? Calculate the variable laplacian >>> >>> ? Sum terms to create RHS (nonlinear advected velocities + >>> laplacian) >>> >>> ? Finally, the runge kutta integration is done in a typical way >>> that looks like: >>> >>> newtemp(t+1) = prevtemp(t) + dt*RHS >>> >>> >>> >>> >>> >>> So my questions are: >>> >>> ? I think my problem is nonlinear, but is being made linearized >>> by the advecting scheme, is this correct? this is to know if i should use >>> the linear or nonlinear routines for TS. >>> >> TSComputeRHSFunctionLinear is just a convenience function for linear >>> ODEs in the form udot=Au. Using it won?t buy you much. So for TS starters, >>> it is fine to assume your problem is nonlinear and think of the form >>> udot=f(t,u) where f is the RHS function. >>> >>> ? How do i know what are the appropriate routines i should be >>> using here? so far i think i should use the following: >>> >>> call TSCreate(PETSC_COMM_WORLD,ts,ierr) >>> >>> call TSSetProblemType(ts,TS_LINEAR,ierr) >>> >>> call TSSetTimeStep(ts,dt,ierr) >>> >>> >>> >>> call TSSetFromOptions(ts,ierr) >>> >>> >>> >>> call TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL,ierr) >>> >>> call TSSolve(ts,loctemperature,ierr) >>> >>> >>> >>> Should i use call TSSetRHSJacobian for the temperature jacobian in >>> this case? >>> >> >>> >> I would suggest to write your own RHSFunction and set it to TS with >>> TSSetRHSFunction(). >>> >> >>> >>> >>> >>> >>> >>> I am using >>> https://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex4.c.html >>> as a general guide, is there a more appropriate example? >>> >> >>> >> ex4 is a good example. In addition, ex9 uses finite volume method >>> with slope limiters to solve a variety of problems such as advection >>> equation, burgers equation and shallow water equation. It might be an >>> overkill, but it seems to be close to the problem you are trying to solve. >>> Note that you might want to use the SSP methods (-ts_type ssp) instead of >>> the classic Runge-Kutta methods for problems with shocks. >>> >> >>> >> Hong (Mr.) >>> >> >>> >>> >>> >>> The dt value and total timesteps are controlled by the model, >>> >>> >>> >>> Thanks for your help, >>> >>> >>> >>> >>> >> >>> > >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 9 22:40:35 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Jul 2019 22:40:35 -0500 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> Message-ID: On Sun, Jul 7, 2019 at 8:48 PM Matthew Knepley wrote: > On Sun, Jul 7, 2019 at 8:37 PM Adrian Croucher > wrote: > >> hi Matt, >> On 29/06/19 12:07 AM, Matthew Knepley wrote: >> >> >> Okay, I ran through this. I am attaching my C version which was easier >> for me to play with in the debugger, but it should not be >> hard to put the few lines into yours. The problem comes from using >> overlap in the redistribution. You can see this by running >> >> mpiexec -n 2 ./distgeom2 -overlap 1 -overlap2 0 >> >> which works fine. If overlap2 is 1, then the face data is shifted. >> >> >> Yes, that's the same thing I found. >> >> Something in DistributeField() does not know that we changed >> the topology after we distributed by adding overlap cells. This is >> strange since DMPlexMigrate() works on the coordinates for >> the overlapped mesh, however they always start at 0 in the vector, so it >> must be that some offset is missing. I am looking for it now. >> >> >> It looked to me like the problem might lie not in DMPlexDistributeField() >> but in DMPlexDistribute(), because if you view the redistribution SF coming >> out of the second call to DMPlexDistribute() it looks wrong. >> >> DMPlexDistributeField() appeared to be doing what it was told, but the SF >> it was given didn't look right to me. >> >> True. I fixed it, but I am just getting the PR in order. Should have it > tomorrow morning. > Sorry this took so long. I put in a PR for this: https://bitbucket.org/petsc/petsc/pull-requests/1858/knepley-fix-plex-distribute-overlap/diff I think it fixes your problem. Thanks, Matt > Thanks, > > Matt > >> - Adrian >> >> -- >> Dr Adrian Croucher >> Senior Research Fellow >> Department of Engineering Science >> University of Auckland, New Zealand >> email: a.croucher at auckland.ac.nz >> tel: +64 (0)9 923 4611 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From maahi.buet at gmail.com Wed Jul 10 17:36:10 2019 From: maahi.buet at gmail.com (Maahi Talukder) Date: Wed, 10 Jul 2019 18:36:10 -0400 Subject: [petsc-users] Making the convergence faster Message-ID: Dear All, I am solving Poisson grid generation equation using 9-point Finite Difference. The type of convergence plot I am getting has been attached herewith. As the plot shows, at the beginning, the convergence is very fast, but later the convergence is slowing down. I am using GMRES with left preconditioning? My question is, is it normal? Any ideas how to make the convergence faster? Thanks, Maahi Talukder -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1.jpg Type: image/jpeg Size: 57853 bytes Desc: not available URL: From jed at jedbrown.org Wed Jul 10 17:40:09 2019 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Jul 2019 16:40:09 -0600 Subject: [petsc-users] Making the convergence faster In-Reply-To: References: Message-ID: <87ftnd1pkm.fsf@jedbrown.org> This is typical for weak preconditioners. Have you tried -pc_type gamg or -pc_type mg (algebraic and geometric multigrid, respectively)? On a structured grid with smooth coefficients, geometric multigrid is possible with low setup cost and convergence in a few iterations independent of problem size. Maahi Talukder via petsc-users writes: > Dear All, > > > I am solving Poisson grid generation equation using 9-point Finite > Difference. The type of convergence plot I am getting has been attached > herewith. As the plot shows, at the beginning, the convergence is very > fast, but later the convergence is slowing down. I am using GMRES with left > preconditioning? > > My question is, is it normal? Any ideas how to make the convergence faster? > > > > Thanks, > Maahi Talukder From a.croucher at auckland.ac.nz Wed Jul 10 21:18:46 2019 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 11 Jul 2019 14:18:46 +1200 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> Message-ID: <717762a0-a68e-3080-ba54-eee76f0f3f7e@auckland.ac.nz> On 10/07/19 3:40 PM, Matthew Knepley wrote: > > Sorry this took so long. I put in a PR for this: > > https://bitbucket.org/petsc/petsc/pull-requests/1858/knepley-fix-plex-distribute-overlap/diff > > I think it fixes your problem. > Thanks very much Matt, that does fix it. Good stuff! - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Wed Jul 10 21:49:13 2019 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 11 Jul 2019 14:49:13 +1200 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: <717762a0-a68e-3080-ba54-eee76f0f3f7e@auckland.ac.nz> References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> <717762a0-a68e-3080-ba54-eee76f0f3f7e@auckland.ac.nz> Message-ID: <34b28ae0-5d40-0cac-8c61-e2d1362bd07a@auckland.ac.nz> Probably don't want the two extra PetscSFView() calls (lines 1687, 1688) though- presumably they were just for temporary debugging? - Adrian On 11/07/19 2:18 PM, Adrian Croucher wrote: > > > On 10/07/19 3:40 PM, Matthew Knepley wrote: >> >> Sorry this took so long. I put in a PR for this: >> >> https://bitbucket.org/petsc/petsc/pull-requests/1858/knepley-fix-plex-distribute-overlap/diff >> >> I think it fixes your problem. >> > Thanks very much Matt, that does fix it. Good stuff! > > - Adrian > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 10 21:54:38 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jul 2019 21:54:38 -0500 Subject: [petsc-users] DMPlexDistributeField In-Reply-To: <34b28ae0-5d40-0cac-8c61-e2d1362bd07a@auckland.ac.nz> References: <991eacaa-f9a5-80e7-abd7-d85e0f310e50@auckland.ac.nz> <11d4323e-eeda-eae9-588c-c7ac85e8b9af@auckland.ac.nz> <3aa2fbb5-adad-93b3-5387-88dcea3fc11f@auckland.ac.nz> <27397288-5c44-6ea6-fdd5-a51a1065c67d@auckland.ac.nz> <717762a0-a68e-3080-ba54-eee76f0f3f7e@auckland.ac.nz> <34b28ae0-5d40-0cac-8c61-e2d1362bd07a@auckland.ac.nz> Message-ID: Crap! The wrong thing got pushed. Thanks, Matt On Wed, Jul 10, 2019 at 9:49 PM Adrian Croucher wrote: > Probably don't want the two extra PetscSFView() calls (lines 1687, 1688) > though- presumably they were just for temporary debugging? > > - Adrian > On 11/07/19 2:18 PM, Adrian Croucher wrote: > > > On 10/07/19 3:40 PM, Matthew Knepley wrote: > > > Sorry this took so long. I put in a PR for this: > > > https://bitbucket.org/petsc/petsc/pull-requests/1858/knepley-fix-plex-distribute-overlap/diff > > I think it fixes your problem. > > Thanks very much Matt, that does fix it. Good stuff! > > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 4611 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From D.Liu-4 at tudelft.nl Thu Jul 11 09:05:37 2019 From: D.Liu-4 at tudelft.nl (Dongyu Liu - CITG) Date: Thu, 11 Jul 2019 14:05:37 +0000 Subject: [petsc-users] What is the best way to do domain decomposition with petsc? Message-ID: <047f7c9745364b0d8cd3b21dbce0598d@tudelft.nl> Hi, We want to incorporate the PETSc into our in-house FEM package. We found two ways to do the domain decomposition. The first one is to read the mesh with partitioning, and the partitioning is done with gmsh. For this one, we need to do the index mapping (renumbering). We found that the mesh nodes cannot be shared by each processor (Is that true?). Then we need to provide the ghosted points information to the PETSc which can be obtained from the gmsh file. To implement this method, we need to provide the local and global mapping like AO in PETSc. We also need to use it with our own mesh data structure. The second way is to use DMPlex to read the mesh and do the partitioning. For this, an interface should be provided to link the DMPlex mesh data structure and to our own mesh object to avoid changing too many places in our codes. I am just wondering which way is better, or do you have any other suggestion? Best, Dongyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 11 09:18:12 2019 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Jul 2019 09:18:12 -0500 Subject: [petsc-users] What is the best way to do domain decomposition with petsc? In-Reply-To: <047f7c9745364b0d8cd3b21dbce0598d@tudelft.nl> References: <047f7c9745364b0d8cd3b21dbce0598d@tudelft.nl> Message-ID: On Thu, Jul 11, 2019 at 9:06 AM Dongyu Liu - CITG via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi, > > > We want to incorporate the PETSc into our in-house FEM package. We found > two ways to do the domain decomposition. > > The first one is to read the mesh with partitioning, and the partitioning > is done with gmsh. For this one, we need to do the index mapping > (renumbering). We found that the mesh nodes cannot be shared by each > processor (Is that true?). > The parallel numbering of unknowns for PETSc (Vec and Mat) needs a contiguous set of non-overlapping indices for each process, so if "nodes" and "unknowns" are the same here, then no nodes cannot be shared. Plex treats these things independently. > Then we need to provide the ghosted points information to the PETSc which > can be obtained from the gmsh file. To implement this method, we need to > provide the local and global mapping like AO in PETSc. > You do need to provide a L2G mapping in order to give PETSc ghost information. I do not understand how this is related to AO, which is a global renumbering. > We also need to use it with our own mesh data structure. > > The second way is to use DMPlex to read the mesh and do the partitioning. > For this, an interface should be provided to link the DMPlex mesh data > structure and to our own mesh object to avoid changing too many places in > our codes. > Yes. I think GMsh just calls a common partitioner (Metis?) so you should be able to replicate it. Also, I think you can use a DMLabel to link the Plex object with your own mesh, meaning you can associate some integer with any part of the mesh you want. > I am just wondering which way is better, or do you have any other > suggestion? > If you plan on doing a lot of mesh manipulation by hand and want to control everything, the first option might be better. On the other hand, if you use Plex, you could potentially take advantage of parallel loading and output, redistribution and load balancing, adaptive refinement, and interfacing with the multilevel and block solvers in PETSc. Thanks, Matt > Best, > > Dongyu > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jul 11 10:02:48 2019 From: jed at jedbrown.org (Jed Brown) Date: Thu, 11 Jul 2019 09:02:48 -0600 Subject: [petsc-users] What is the best way to do domain decomposition with petsc? In-Reply-To: References: <047f7c9745364b0d8cd3b21dbce0598d@tudelft.nl> Message-ID: <87tvbszk9z.fsf@jedbrown.org> Matthew Knepley via petsc-users writes: >> I am just wondering which way is better, or do you have any other >> suggestion? >> > If you plan on doing a lot of mesh manipulation by hand and want to control > everything, the first option might be better. > On the other hand, if you use Plex, you could potentially take advantage of > parallel loading and output, redistribution > and load balancing, adaptive refinement, and interfacing with the > multilevel and block solvers in PETSc. There's also a workflow factor. If you rely on partitioning via Gmsh as a preprocessing step, then you can't easily run the same mesh on a different number of processes. With DMPlex doing the partitioning and distribution automatically on load (it's fast), you can much more easily conduct scaling studies, debug configurations, move to different machines, etc. From mpovolot at purdue.edu Thu Jul 11 12:36:07 2019 From: mpovolot at purdue.edu (Povolotskyi, Mykhailo) Date: Thu, 11 Jul 2019 17:36:07 +0000 Subject: [petsc-users] problem with CISS in SLEPC Message-ID: Hello, I want to use CISS for my eigenvalue problem. To test it I tried a simple (2x2) matrix case, but the code failed with the following message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in LAPACK subroutine gesvd: info=127 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 [0]PETSC ERROR: var1 on a linux-complex named brown-a027.rcac.purdue.edu by mpovolot Thu Jul 11 13:28:21 2019 [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-x=0 --with-hdf5 --download-hdf5=1 --with-single-library=1 --with-pic=1 --with-shared-libraries=0 --with-log=0 --with-clanguage=C++ --CXXFLAGS="-fopenmp -fPIC" --CFLAGS="-fopenmp -fPIC" --with-fortran=0 --FFLAGS="-fopenmp -fPIC" --with-debugging=0 --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS= CXXOPTFLAGS= FOPTFLAGS= --download-metis=1 --download-parmetis=1 --with-valgrind-dir=/apps/brown/valgrind/3.13.0_gcc-4.8.5 --download-mumps=1 --with-fortran-kernels=0 --download-superlu_dist=1 --with-blaslapack-lib="-L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core " --with-blacs-lib=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include --with-scalapack-lib="-Wl,-rpath,/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core? -lpthread -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include [0]PETSC ERROR: #1 SVD_S() line 611 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c [0]PETSC ERROR: #2 EPSSolve_CISS() line 1028 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c [0]PETSC ERROR: #3 EPSSolve() line 147 in /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/interface/epssolve.c Here is the code: void test1() { ? Mat matrix_petsc; MatCreateDense(MPI_COMM_SELF,2,2,PETSC_DECIDE,PETSC_DECIDE,NULL,&matrix_petsc); MatSetValue(matrix_petsc,0,0,complex(1,0),INSERT_VALUES); MatSetValue(matrix_petsc,1,1,complex(2,0),INSERT_VALUES); MatSetValue(matrix_petsc,0,1,complex(0,0),INSERT_VALUES); MatSetValue(matrix_petsc,1,0,complex(0,0),INSERT_VALUES); ? MatAssemblyBegin(matrix_petsc, MAT_FINAL_ASSEMBLY); ? MatAssemblyEnd(matrix_petsc, MAT_FINAL_ASSEMBLY); ?{ ????? EPS??????????? eps; ????? EPSType??????? type; ????? RG???????????? rg; ????? EPSCreate(MPI_COMM_SELF,&eps); ????? EPSSetOperators( eps,matrix_petsc,NULL); ????? EPSSetType(eps,EPSCISS); ????? EPSSetProblemType(eps, EPS_NHEP); ????? double real_min = -10; ????? double real_max =? 10; ????? double imag_min = -10; ????? double imag_max = 10; ????? EPSGetRG(eps,&rg); ????? RGSetType(rg,RGINTERVAL); RGIntervalSetEndpoints(rg,real_min,real_max,imag_min,imag_max); ????? EPSSolve(eps); ????? EPSDestroy(&eps); ? } ? MatDestroy(&matrix_petsc); } Do you see anything wrong in the way I'm calling SLEPC functions? Thank you, Michael. From knepley at gmail.com Thu Jul 11 12:43:25 2019 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Jul 2019 12:43:25 -0500 Subject: [petsc-users] problem with CISS in SLEPC In-Reply-To: References: Message-ID: On Thu, Jul 11, 2019 at 12:37 PM Povolotskyi, Mykhailo via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I want to use CISS for my eigenvalue problem. To test it I tried a > simple (2x2) matrix case, but the code failed with the following message: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in LAPACK subroutine gesvd: info=127 > This does not make sense. The error is supposed to be the number of non-converged diagonals, but you only have a 2x2 problem, so something else is wrong. Matt > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 > [0]PETSC ERROR: var1 on a linux-complex named brown-a027.rcac.purdue.edu > by mpovolot Thu Jul 11 13:28:21 2019 > [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-x=0 > --with-hdf5 --download-hdf5=1 --with-single-library=1 --with-pic=1 > --with-shared-libraries=0 --with-log=0 --with-clanguage=C++ > --CXXFLAGS="-fopenmp -fPIC" --CFLAGS="-fopenmp -fPIC" --with-fortran=0 > --FFLAGS="-fopenmp -fPIC" --with-debugging=0 --with-cc=mpicc > --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS= CXXOPTFLAGS= FOPTFLAGS= > --download-metis=1 --download-parmetis=1 > --with-valgrind-dir=/apps/brown/valgrind/3.13.0_gcc-4.8.5 > --download-mumps=1 --with-fortran-kernels=0 --download-superlu_dist=1 > --with-blaslapack-lib="-L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > > -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core " > --with-blacs-lib=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so > > --with-blacs-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include > > --with-scalapack-lib="-Wl,-rpath,/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > > -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > > -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread > -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > > -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" > > --with-scalapack-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include > [0]PETSC ERROR: #1 SVD_S() line 611 in > > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c > [0]PETSC ERROR: #2 EPSSolve_CISS() line 1028 in > > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c > [0]PETSC ERROR: #3 EPSSolve() line 147 in > > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/interface/epssolve.c > > > Here is the code: > > void test1() > { > Mat matrix_petsc; > > MatCreateDense(MPI_COMM_SELF,2,2,PETSC_DECIDE,PETSC_DECIDE,NULL,&matrix_petsc); > MatSetValue(matrix_petsc,0,0,complex(1,0),INSERT_VALUES); > MatSetValue(matrix_petsc,1,1,complex(2,0),INSERT_VALUES); > MatSetValue(matrix_petsc,0,1,complex(0,0),INSERT_VALUES); > MatSetValue(matrix_petsc,1,0,complex(0,0),INSERT_VALUES); > MatAssemblyBegin(matrix_petsc, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(matrix_petsc, MAT_FINAL_ASSEMBLY); > > { > EPS eps; > EPSType type; > RG rg; > > EPSCreate(MPI_COMM_SELF,&eps); > EPSSetOperators( eps,matrix_petsc,NULL); > EPSSetType(eps,EPSCISS); > EPSSetProblemType(eps, EPS_NHEP); > > double real_min = -10; > double real_max = 10; > > double imag_min = -10; > double imag_max = 10; > > EPSGetRG(eps,&rg); > RGSetType(rg,RGINTERVAL); > RGIntervalSetEndpoints(rg,real_min,real_max,imag_min,imag_max); > > EPSSolve(eps); > > EPSDestroy(&eps); > } > > > MatDestroy(&matrix_petsc); > > } > > > Do you see anything wrong in the way I'm calling SLEPC functions? > > Thank you, > > Michael. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Jul 11 12:56:16 2019 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 11 Jul 2019 19:56:16 +0200 Subject: [petsc-users] problem with CISS in SLEPC In-Reply-To: References: Message-ID: <3807B224-14FE-4530-9694-62AE2CFD68E4@dsic.upv.es> Internally the solver builds a subspace of dimension L*M, which defaults to 128. See the details here http://slepc.upv.es/documentation/reports/str11.pdf I guess the method is not prepared for smallish matrices. Try reducing the values of L and M. Jose > El 11 jul 2019, a las 19:43, Matthew Knepley via petsc-users escribi?: > > On Thu, Jul 11, 2019 at 12:37 PM Povolotskyi, Mykhailo via petsc-users wrote: > Hello, > > I want to use CISS for my eigenvalue problem. To test it I tried a > simple (2x2) matrix case, but the code failed with the following message: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in LAPACK subroutine gesvd: info=127 > > This does not make sense. The error is supposed to be the number of non-converged diagonals, but > you only have a 2x2 problem, so something else is wrong. > > Matt > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 > [0]PETSC ERROR: var1 on a linux-complex named brown-a027.rcac.purdue.edu > by mpovolot Thu Jul 11 13:28:21 2019 > [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-x=0 > --with-hdf5 --download-hdf5=1 --with-single-library=1 --with-pic=1 > --with-shared-libraries=0 --with-log=0 --with-clanguage=C++ > --CXXFLAGS="-fopenmp -fPIC" --CFLAGS="-fopenmp -fPIC" --with-fortran=0 > --FFLAGS="-fopenmp -fPIC" --with-debugging=0 --with-cc=mpicc > --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS= CXXOPTFLAGS= FOPTFLAGS= > --download-metis=1 --download-parmetis=1 > --with-valgrind-dir=/apps/brown/valgrind/3.13.0_gcc-4.8.5 > --download-mumps=1 --with-fortran-kernels=0 --download-superlu_dist=1 > --with-blaslapack-lib="-L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core " > --with-blacs-lib=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so > --with-blacs-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include > --with-scalapack-lib="-Wl,-rpath,/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread > -L/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64 > -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" > --with-scalapack-include=/apps/cent7/intel/compilers_and_libraries_2017.1.132/linux/mkl/include > [0]PETSC ERROR: #1 SVD_S() line 611 in > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c > [0]PETSC ERROR: #2 EPSSolve_CISS() line 1028 in > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/impls/ciss/ciss.c > [0]PETSC ERROR: #3 EPSSolve() line 147 in > /depot/kildisha/apps/brown/nemo5/libs/slepc/build-cplx/src/eps/interface/epssolve.c > > > Here is the code: > > void test1() > { > Mat matrix_petsc; > MatCreateDense(MPI_COMM_SELF,2,2,PETSC_DECIDE,PETSC_DECIDE,NULL,&matrix_petsc); > MatSetValue(matrix_petsc,0,0,complex(1,0),INSERT_VALUES); > MatSetValue(matrix_petsc,1,1,complex(2,0),INSERT_VALUES); > MatSetValue(matrix_petsc,0,1,complex(0,0),INSERT_VALUES); > MatSetValue(matrix_petsc,1,0,complex(0,0),INSERT_VALUES); > MatAssemblyBegin(matrix_petsc, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(matrix_petsc, MAT_FINAL_ASSEMBLY); > > { > EPS eps; > EPSType type; > RG rg; > > EPSCreate(MPI_COMM_SELF,&eps); > EPSSetOperators( eps,matrix_petsc,NULL); > EPSSetType(eps,EPSCISS); > EPSSetProblemType(eps, EPS_NHEP); > > double real_min = -10; > double real_max = 10; > > double imag_min = -10; > double imag_max = 10; > > EPSGetRG(eps,&rg); > RGSetType(rg,RGINTERVAL); > RGIntervalSetEndpoints(rg,real_min,real_max,imag_min,imag_max); > > EPSSolve(eps); > > EPSDestroy(&eps); > } > > > MatDestroy(&matrix_petsc); > > } > > > Do you see anything wrong in the way I'm calling SLEPC functions? > > Thank you, > > Michael. > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From dayedut123 at 163.com Thu Jul 11 19:45:53 2019 From: dayedut123 at 163.com (=?GBK?B?ztI=?=) Date: Fri, 12 Jul 2019 08:45:53 +0800 (CST) Subject: [petsc-users] PETSC ERROR: Petsc has generated inconsistent data with PCGAMG and KSPGRMES In-Reply-To: <4DDB22E9-1A8E-45A7-9919-C9063D8C38A0@anl.gov> References: <3075ad67.12c81.16bd6ede824.Coremail.dayedut123@163.com> <4DDB22E9-1A8E-45A7-9919-C9063D8C38A0@anl.gov> Message-ID: <5ae6210c.32ae.16be3a4be7f.Coremail.dayedut123@163.com> Thank you very much! Daye At 2019-07-10 02:36:55, "Smith, Barry F." wrote: > > Most preconditioners don't care if the matrix has non-symmetric non-zero structure. GAMG requires it have a symmetric nonzero structure. It lists some options you can try to make it symmetric. > > >> [2]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. > > > >> On Jul 9, 2019, at 8:30 AM, ? via petsc-users wrote: >> >> Hello all, >> I set PCGAMG and KSPGRMES to solve the matrix which is formed by a particle method such as SPH. But when there is always a PETSC ERROR: >> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [2]PETSC ERROR: Petsc has generated inconsistent data >> [2]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric. >> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 >> [2]PETSC ERROR: ./MPS on a arch-linux2-cxx-debug named daye-Precision-7920-Tower by daye Tue Jul 9 20:59:39 2019 >> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-hypre --with-clanguage=cxx >> [2]PETSC ERROR: #1 smoothAggs() line 418 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c >> [2]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 970 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/agg.c >> [2]PETSC ERROR: #3 PCSetUp_GAMG() line 518 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/impls/gamg/gamg.c >> [2]PETSC ERROR: #4 PCSetUp() line 923 in /home/daye/PETSC/petsc-3.9.1/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: #5 KSPSetUp() line 381 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: #6 KSPSolve() line 612 in /home/daye/PETSC/petsc-3.9.1/src/ksp/ksp/interface/itfunc.c >> ^C[mpiexec at daye-Precision-7920-Tower] Sending Ctrl-C to processes as requested >> [mpiexec at daye-Precision-7920-Tower] Press Ctrl-C again to force abort >> >> This matrix can be solved when I use PCHYPRE and KSPGRMES. But when I change the pre-condition to PCGAMG, this error always occurs. Any suggestions about that? >> Thank you at first! >> >> Daye >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Fri Jul 12 05:12:35 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Fri, 12 Jul 2019 19:12:35 +0900 Subject: [petsc-users] Various Questions Regarding PETSC Message-ID: Hello all, I have a few question regarding Petsc, Question 1: For the profiling , is it possible to only show the user defined log events in the breakdown of each stage in Log-view. I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, PetscLogEventExcludeClass(MAT_CLASSID); PetscLogEventExcludeClass(VEC_CLASSID); PetscLogEventExcludeClass(KSP_CLASSID); PetscLogEventExcludeClass(PC_CLASSID); which should "Deactivates event logging for a PETSc object class in every stage" according to the manual. however I still see them in the stage breakdown --- Event Stage 1: Matrix Construction BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT Also is possible to clear the logs so that I can write a separate profiling output file for each timestep ( since I am solving a transient problem and I want to know the change in performance as time goes by ) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Question 2: Regarding MatSetValues Right now, I writing a finite volume code, due to algorithm requirement I have to write the matrix into local native format ( array of arrays) and then loop through rows and use MatSetValues to set the elements in "Mat A" MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); but it is very slow and it is killing my performance although the matrix was properly set using MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, PETSC_DETERMINE, PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); with d_nnz,and o_nnz properly assigned so no mallocs occur during matsetvalues and all inserted values are local so no off-processor values So my question is it possible to set multiple rows at once hopefully all, I checked the manual and MatSetValues can only set dense matrix block because it seems that row by row is expensive Or perhaps is it possible to copy all rows to the underlying matrix data, as I mentioned all values are local and no off-processor values ( stash is 0 ) [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Question 3: If all matrix and vector inserted data are local, what part of the vec/mat assembly consumes time because matsetvalues and matassembly consume more time than matrix builder Also this is not just for the first time MAT_FINAL_ASSEMBLY For context the matrix in the above is nearly 1Mx1M partitioned over six processes and it was NOT built using DM Finally the configure options are: Configure options: PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre Sorry for such long question and thanks in advance Thanks M. Kamra -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 12 07:23:19 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Jul 2019 07:23:19 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello all, > I have a few question regarding Petsc, > Please send the entire output of a run with all the logging turned on, using -log_view and -info. Thanks, Matt > Question 1: > For the profiling , is it possible to only show the user defined log > events in the breakdown of each stage in Log-view. > I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, > PetscLogEventExcludeClass(MAT_CLASSID); > PetscLogEventExcludeClass(VEC_CLASSID); > PetscLogEventExcludeClass(KSP_CLASSID); > PetscLogEventExcludeClass(PC_CLASSID); > which should "Deactivates event logging for a PETSc object class in every > stage" according to the manual. > however I still see them in the stage breakdown > --- Event Stage 1: Matrix Construction > > BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 > 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 > VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 > 8.0e+00 0 0 3 0 6 10 0 50 20100 0 > AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 > 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT > myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT > setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT > setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT > callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT > > Also is possible to clear the logs so that I can write a separate > profiling output file for each timestep ( since I am solving a transient > problem and I want to know the change in performance as time goes by ) > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Question 2: > Regarding MatSetValues > Right now, I writing a finite volume code, due to algorithm requirement I > have to write the matrix into local native format ( array of arrays) and > then loop through rows and use MatSetValues to set the elements in "Mat A" > MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); > but it is very slow and it is killing my performance > although the matrix was properly set using > MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, > PETSC_DETERMINE, > PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); > with d_nnz,and o_nnz properly assigned so no mallocs occur during > matsetvalues and all inserted values are local so no off-processor values > So my question is it possible to set multiple rows at once hopefully all, > I checked the manual and MatSetValues can only set dense matrix block > because it seems that row by row is expensive > Or perhaps is it possible to copy all rows to the underlying matrix data, > as I mentioned all values are local and no off-processor values ( stash is > 0 ) > [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. > [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: > 0 unneeded,743028 used > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: > 0 unneeded,742972 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: > 0 unneeded,743093 used > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: > 0 unneeded,743036 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: > 0 unneeded,742938 used > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: > 0 unneeded,743049 used > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 > unneeded,685 used > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 > unneeded,649 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 > unneeded,1011 used > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 > unneeded,1137 used > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 > unneeded,658 used > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 > unneeded,648 used > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Question 3: > If all matrix and vector inserted data are local, what part of the vec/mat > assembly consumes time because matsetvalues and matassembly consume more > time than matrix builder > Also this is not just for the first time MAT_FINAL_ASSEMBLY > > > For context the matrix in the above is nearly 1Mx1M partitioned over six > processes and it was NOT built using DM > > Finally the configure options are: > > Configure options: > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native > -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 > -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx > --with-fc=mpif90 --download-metis --download-hypre > > Sorry for such long question and thanks in advance > Thanks > M. Kamra > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Fri Jul 12 14:41:12 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Fri, 12 Jul 2019 12:41:12 -0700 Subject: [petsc-users] Reshape a 3D array into Matrix? Message-ID: Hi petsc devs and users, Is there an analogue to reshape as in the fortran / matlab functions for petsc arrays? i am looking into converting a 3D DMDA array of (nx,ny,nz) sizes into a 2D MATMPI matrix of size (nx) x (ny) x (nz) by (nx) x (ny) x (nz), that will be used as the laplacian matrix in a linear solver, which will change in every iteration, so is easier to calculate in 3D and convert. Thanks for your help, -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 12 16:16:07 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Jul 2019 16:16:07 -0500 Subject: [petsc-users] Reshape a 3D array into Matrix? In-Reply-To: References: Message-ID: On Fri, Jul 12, 2019 at 2:43 PM Manuel Valera via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi petsc devs and users, > > Is there an analogue to reshape as in the fortran / matlab functions for > petsc arrays? i am looking into converting a 3D DMDA array of (nx,ny,nz) > sizes into a 2D MATMPI matrix of size (nx) x (ny) x (nz) by (nx) x (ny) x > (nz), that will be used as the laplacian matrix in a linear solver, which > will change in every iteration, so is easier to calculate in 3D and convert. > I do not think this makes sense. A DMDA array is a Vec, or function on the grid. A Mat is a linear operator between those functions. Thanks, Matt > Thanks for your help, > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Fri Jul 12 22:51:46 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sat, 13 Jul 2019 12:51:46 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Hello Matt, Attached is the dumped entire log output using -log_view and -info. Thanks, Kamra On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley wrote: > On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello all, >> I have a few question regarding Petsc, >> > > Please send the entire output of a run with all the logging turned on, > using -log_view and -info. > > Thanks, > > Matt > > >> Question 1: >> For the profiling , is it possible to only show the user defined log >> events in the breakdown of each stage in Log-view. >> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >> PetscLogEventExcludeClass(MAT_CLASSID); >> PetscLogEventExcludeClass(VEC_CLASSID); >> PetscLogEventExcludeClass(KSP_CLASSID); >> PetscLogEventExcludeClass(PC_CLASSID); >> which should "Deactivates event logging for a PETSc object class in every >> stage" according to the manual. >> however I still see them in the stage breakdown >> --- Event Stage 1: Matrix Construction >> >> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 >> 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 >> 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 >> 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >> >> Also is possible to clear the logs so that I can write a separate >> profiling output file for each timestep ( since I am solving a transient >> problem and I want to know the change in performance as time goes by ) >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> Question 2: >> Regarding MatSetValues >> Right now, I writing a finite volume code, due to algorithm requirement I >> have to write the matrix into local native format ( array of arrays) and >> then loop through rows and use MatSetValues to set the elements in "Mat A" >> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >> but it is very slow and it is killing my performance >> although the matrix was properly set using >> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >> PETSC_DETERMINE, >> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >> with d_nnz,and o_nnz properly assigned so no mallocs occur during >> matsetvalues and all inserted values are local so no off-processor values >> So my question is it possible to set multiple rows at once hopefully all, >> I checked the manual and MatSetValues can only set dense matrix block >> because it seems that row by row is expensive >> Or perhaps is it possible to copy all rows to the underlying matrix data, >> as I mentioned all values are local and no off-processor values ( stash is >> 0 ) >> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. >> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: >> 0 unneeded,743028 used >> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: >> 0 unneeded,742972 used >> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: >> 0 unneeded,743093 used >> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: >> 0 unneeded,743036 used >> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: >> 0 unneeded,742938 used >> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: >> 0 unneeded,743049 used >> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 >> unneeded,685 used >> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 >> unneeded,649 used >> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 >> unneeded,1011 used >> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 >> unneeded,1137 used >> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 >> unneeded,658 used >> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 >> unneeded,648 used >> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> Question 3: >> If all matrix and vector inserted data are local, what part of the >> vec/mat assembly consumes time because matsetvalues and matassembly consume >> more time than matrix builder >> Also this is not just for the first time MAT_FINAL_ASSEMBLY >> >> >> For context the matrix in the above is nearly 1Mx1M partitioned over six >> processes and it was NOT built using DM >> >> Finally the configure options are: >> >> Configure options: >> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >> --with-fc=mpif90 --download-metis --download-hypre >> >> Sorry for such long question and thanks in advance >> Thanks >> M. Kamra >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0] PetscInitialize(): PETSc successfully started: number of processors = 6 [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [1] PetscInitialize(): PETSc successfully started: number of processors = 6 [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [2] PetscInitialize(): PETSc successfully started: number of processors = 6 [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [3] PetscInitialize(): PETSc successfully started: number of processors = 6 [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [4] PetscInitialize(): PETSc successfully started: number of processors = 6 [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [5] PetscInitialize(): PETSc successfully started: number of processors = 6 [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 Processor: 0/6 [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 268435455 [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [2] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [3] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [4] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [5] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [3] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [2] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [4] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 268435455 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 3 scatter [0] VecScatterCreate_PtoS(): Special case: blocked indices to stride [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 <<< Linear Solver Setup Successfully >>> [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 3 scatter [0] VecScatterCreate_PtoS(): Special case: blocked indices to stride [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 3 scatter [0] VecScatterCreate_PtoS(): Special case: blocked indices to stride [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 Set Matrix Start [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatSeqAIJCheckInode(): Found 186063 nodes out of 186063 rows. Not using Inode routines [2] MatSeqAIJCheckInode(): Found 186064 nodes out of 186064 rows. Not using Inode routines [5] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [0] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [4] MatSeqAIJCheckInode(): Found 186063 nodes out of 186063 rows. Not using Inode routines [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. Time to Setup (II): 5.26303 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. Time to Setup (III): 4.77501, count = 1 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [5] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [1] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [2] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [3] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [4] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 930313; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 930313; storage space: 0 unneeded,649 used [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,648 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 930312; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,1011 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [3] MatSeqAIJCheckInode(): Found 37997 nodes of 186063. Limit used: 5. Using Inode routines [4] MatSeqAIJCheckInode(): Found 37991 nodes of 186063. Limit used: 5. Using Inode routines [2] MatSeqAIJCheckInode(): Found 38030 nodes of 186064. Limit used: 5. Using Inode routines [0] MatSeqAIJCheckInode(): Found 37993 nodes of 186062. Limit used: 5. Using Inode routines [1] MatSeqAIJCheckInode(): Found 38413 nodes of 186062. Limit used: 5. Using Inode routines [5] MatSeqAIJCheckInode(): Found 38565 nodes of 186062. Limit used: 5. Using Inode routines [2] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930327 [3] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930328 [0] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [1] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [5] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [4] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930328 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 930312 X 186064; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 929627)/(num_localrows 930312) > 0.6. Use CompressedRow routines. [2] MatSeqAIJCheckInode(): Found 186870 nodes of 930312. Limit used: 5. Using Inode routines [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 929177)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [5] MatSeqAIJCheckInode(): Found 187410 nodes of 930314. Limit used: 5. Using Inode routines [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 929303)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [1] MatSeqAIJCheckInode(): Found 187283 nodes of 930314. Limit used: 5. Using Inode routines [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 930313 X 186063; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 929655)/(num_localrows 930313) > 0.6. Use CompressedRow routines. [3] MatSeqAIJCheckInode(): Found 186842 nodes of 930313. Limit used: 5. Using Inode routines [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 929666)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [0] MatSeqAIJCheckInode(): Found 186837 nodes of 930314. Limit used: 5. Using Inode routines isMatSymm = true Set Matrix Done Setting Preconditioner DONE Setting Iterative Solver DONE [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 930313 X 186063; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 929664)/(num_localrows 930313) > 0.6. Use CompressedRow routines. [4] MatSeqAIJCheckInode(): Found 186825 nodes of 930313. Limit used: 5. Using Inode routines [0] PCSetUp(): Setting up PC for first time [0] MatConvert(): Check superclass seqhypre mpiaij -> 0 [0] MatConvert(): Check superclass mpihypre mpiaij -> 0 [0] MatConvert(): Check specialized (1) MatConvert_mpiaij_seqhypre_C (mpiaij) -> 0 [0] MatConvert(): Check specialized (1) MatConvert_mpiaij_mpihypre_C (mpiaij) -> 0 [0] MatConvert(): Check specialized (1) MatConvert_mpiaij_hypre_C (mpiaij) -> 1 [0] KSPConvergedDefault(): user has provided nonzero initial guess, computing 2-norm of RHS [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] KSPConvergedDefault(): Linear solver has converged. Residual norm 6.843645053482e-13 is less than absolute tolerance 1.000000000000e-12 at iteration 17 Solution Converged in 17 to error 6.84365e-13 where tol = 1.00000e-05 Init Res = 1. , Converged @@ 17 iterations and the reason is ATOL and resNorm = 6.8436e-13[0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. Updating Auxilary Field Face Data (-1.00243, 1.00399, 112999.) LeastSquare Gradient is Happy [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. LeastSquare Gradient is Happy [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. ------------------------------------------------------------------------------ Variable : Generic Variable (none) -------------------------------------------------------------- Variable is computed at the cell Center isCellData Up2Date : yes isFaceData Up2Date : yes isGradient Up2Date : yes ------------------------------------------------------------------------------ Writing File <OutputData/VTK/PoissonBench000000.pvtu> [4] PetscFinalize(): PetscFinalize() called [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscFinalize(): PetscFinalize() called [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscFinalize(): PetscFinalize() called [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscFinalize(): PetscFinalize() called [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscFinalize(): PetscFinalize() called [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscFinalize(): PetscFinalize() called [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ./OpenAqua300 on a release3 named mohamedkamra-DAIV-DQZ520 with 6 processors, by mohamedkamra Sat Jul 13 12:32:48 2019 Using Petsc Release Version 3.11.2, May, 18, 2019 Max Max/Min Avg Total [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) Time (sec): 2.257e+01 1.000 2.257e+01 Objects: 7.000e+01 1.000 7.000e+01 Flop: 6.260e+07 1.000 6.260e+07 3.756e+08 Flop/sec: 2.774e+06 1.000 2.773e+06 1.664e+07 MPI Messages: 1.255e+03 2.427 7.630e+02 4.578e+03 MPI Message Lengths: 5.430e+07 1.018 7.029e+04 3.218e+08 MPI Reductions: 1.410e+02 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 8.6126e+00 38.2% 0.0000e+00 0.0% 4.140e+02 9.0% 1.713e+03 0.2% 6.200e+01 44.0% 1: Matrix Construction: 1.0037e+01 44.5% 0.0000e+00 0.0% 3.636e+03 79.4% 2.112e+03 2.4% 8.000e+00 5.7% 2: System Solution: 3.6352e+00 16.1% 3.7166e+08 99.0% 4.920e+02 10.7% 6.365e+05 97.3% 6.400e+01 45.4% 3: Update AuxData: 2.8637e-01 1.3% 3.9110e+06 1.0% 3.600e+01 0.8% 6.384e+03 0.1% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 9 1.0 6.8691e-02255.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 17 1.0 4.3519e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 18 1.0 6.8860e-02198.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 18 1.0 3.3617e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 9 1.0 3.2783e-04 2.1 0.00e+00 0.0 1.6e+02 3.5e+03 0.0e+00 0 0 4 0 0 0 0 39 81 0 0 VecScatterEnd 9 1.0 3.1114e-04 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: Matrix Construction BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 --- Event Stage 2: System Solution BuildTwoSidedF 2 1.0 3.1161e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecTDot 34 1.0 2.9358e-02 1.7 1.27e+07 1.0 0.0e+00 0.0e+00 3.4e+01 0 20 0 0 24 1 20 0 0 53 2586 VecNorm 19 1.0 8.2438e-03 1.1 7.07e+06 1.0 0.0e+00 0.0e+00 1.9e+01 0 11 0 0 13 0 11 0 0 30 5146 VecCopy 1 1.0 3.7718e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 17 1.0 6.2864e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 34 1.0 2.3406e-02 1.0 1.27e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 20 0 0 0 1 20 0 0 0 3243 VecAYPX 17 1.0 1.0511e-02 1.1 6.14e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 3505 VecAssemblyBegin 1 1.0 1.4949e-04 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 1 1.0 3.0994e-06 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 19 1.0 7.8726e-04 1.7 0.00e+00 0.0 3.4e+02 2.1e+03 0.0e+00 0 0 7 0 0 0 0 70 0 0 0 VecScatterEnd 19 1.0 2.5105e-0311.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 18 1.0 7.5983e-02 1.0 2.34e+07 1.0 3.2e+02 2.1e+03 0.0e+00 0 37 7 0 0 2 38 66 0 0 1850 MatConvert 1 1.0 5.0359e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 3 1 0 0 0 6 0 MatAssemblyBegin 3 1.0 1.7092e-0344.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 4.8217e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 3 1 0 0 0 6 0 MatGetRowIJ 2 1.0 3.8147e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 2 1.0 1.1497e+00 1.6 0.00e+00 0.0 1.5e+02 2.1e+06 4.0e+00 4 0 3 97 3 25 0 30100 6 0 KSPSetUp 1 1.0 4.1850e-01211.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 1 0 0 0 1 6 0 0 0 3 0 KSPSolve 1 1.0 1.3674e+00 1.0 6.20e+07 1.0 3.2e+02 2.1e+03 5.3e+01 6 99 7 0 38 38100 66 0 83 272 PCSetUp 1 1.0 1.0755e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 5 0 0 0 3 30 0 0 0 6 0 PCApply 17 1.0 1.2312e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 34 0 0 0 0 0 --- Event Stage 3: Update AuxData BuildTwoSidedF 2 1.0 2.6415e-0228.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 6 0 0 0 0 0 VecMax 1 1.0 1.0192e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMin 1 1.0 1.0574e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNorm 1 1.0 9.1124e-04 1.0 6.52e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0100 0 0 0 4292 VecAssemblyBegin 3 1.0 2.6719e-0228.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 6 0 0 0 0 0 VecAssemblyEnd 3 1.0 8.8215e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 1.1158e-04 2.6 0.00e+00 0.0 3.6e+01 6.4e+03 0.0e+00 0 0 1 0 0 0 0100100 0 0 VecScatterEnd 2 1.0 2.3389e-0414.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 20 0 0 0. Index Set 14 14 29232 0. IS L to G Mapping 7 0 0 0. Vec Scatter 7 0 0 0. Matrix 3 0 0 0. Krylov Solver 1 0 0 0. Viewer 1 0 0 0. --- Event Stage 1: Matrix Construction Vector 2 1 1664 0. Index Set 2 2 4176 0. Vec Scatter 1 0 0 0. --- Event Stage 2: System Solution Vector 6 3 1493488 0. Index Set 2 2 3722840 0. Matrix 3 2 22349512 0. Preconditioner 1 0 0 0. --- Event Stage 3: Update AuxData ======================================================================================================================== Average time to get PetscTime(): 8.10623e-07 Average time for MPI_Barrier(): 2.09808e-06 Average time for zero size MPI_Send(): 8.34465e-07 #PETSc Option Table entries: -info -ksp_monitor -log_view -pc_hypre_boomeramg_max_iter 1 -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_rtol 0 -pc_hypre_boomeramg_strong_threshold 0.4 [1] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [2] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [3] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [4] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [5] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 -pc_pfmg_num_post_relax 1 -pc_pfmg_num_pre_relax 2 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre ----------------------------------------- Libraries compiled on 2019-07-03 09:57:26 on mohamedkamra-DAIV-DQZ520 Machine characteristics: Linux-4.18.0-25-generic-x86_64-with-Ubuntu-18.04-bionic Using PETSc directory: /home/mohamedkamra/Downloads/petsc-3.11.2 Using PETSc arch: release3 ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 -march=native -mtune=native Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -march=native -mtune=native ----------------------------------------- Using include paths: -I/home/mohamedkamra/Downloads/petsc-3.11.2/include -I/home/mohamedkamra/Downloads/petsc-3.11.2/release3/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -lpetsc -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lHYPRE -llapack -lblas -lmetis -lm -lX11 -lpthread -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl ----------------------------------------- [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374780 From knepley at gmail.com Fri Jul 12 23:42:48 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Jul 2019 23:42:48 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa wrote: > Hello Matt, > Attached is the dumped entire log output using -log_view and -info. > In matrix construction, it looks like you have a mixture of load imbalance (see the imbalance in the Begin events) and lots of Scatter messages in your assembly. We turn off MatSetValues() logging by default since it is usually called many times, but you can explicitly turn it back on if you want. I don't think that is the problem here. Its easy to see from examples (say SNES ex5) that it is not the major time sink. What is the Scatter doing? Thanks, Matt > Thanks, > Kamra > > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley wrote: > >> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello all, >>> I have a few question regarding Petsc, >>> >> >> Please send the entire output of a run with all the logging turned on, >> using -log_view and -info. >> >> Thanks, >> >> Matt >> >> >>> Question 1: >>> For the profiling , is it possible to only show the user defined log >>> events in the breakdown of each stage in Log-view. >>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>> PetscLogEventExcludeClass(MAT_CLASSID); >>> PetscLogEventExcludeClass(VEC_CLASSID); >>> PetscLogEventExcludeClass(KSP_CLASSID); >>> PetscLogEventExcludeClass(PC_CLASSID); >>> which should "Deactivates event logging for a PETSc object class in >>> every stage" according to the manual. >>> however I still see them in the stage breakdown >>> --- Event Stage 1: Matrix Construction >>> >>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 >>> 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 >>> 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 >>> 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>> >>> Also is possible to clear the logs so that I can write a separate >>> profiling output file for each timestep ( since I am solving a transient >>> problem and I want to know the change in performance as time goes by ) >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> Question 2: >>> Regarding MatSetValues >>> Right now, I writing a finite volume code, due to algorithm requirement >>> I have to write the matrix into local native format ( array of arrays) and >>> then loop through rows and use MatSetValues to set the elements in "Mat A" >>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>> but it is very slow and it is killing my performance >>> although the matrix was properly set using >>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>> PETSC_DETERMINE, >>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>> matsetvalues and all inserted values are local so no off-processor values >>> So my question is it possible to set multiple rows at once hopefully >>> all, I checked the manual and MatSetValues can only set dense matrix block >>> because it seems that row by row is expensive >>> Or perhaps is it possible to copy all rows to the underlying matrix >>> data, as I mentioned all values are local and no off-processor values ( >>> stash is 0 ) >>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>> mallocs. >>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>> space: 0 unneeded,743028 used >>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742972 used >>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743093 used >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,743036 used >>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742938 used >>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743049 used >>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 >>> unneeded,685 used >>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 >>> unneeded,649 used >>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: >>> 0 unneeded,1011 used >>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: >>> 0 unneeded,1137 used >>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 >>> unneeded,658 used >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 >>> unneeded,648 used >>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> Question 3: >>> If all matrix and vector inserted data are local, what part of the >>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>> more time than matrix builder >>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>> >>> >>> For context the matrix in the above is nearly 1Mx1M partitioned over six >>> processes and it was NOT built using DM >>> >>> Finally the configure options are: >>> >>> Configure options: >>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>> --with-fc=mpif90 --download-metis --download-hypre >>> >>> Sorry for such long question and thanks in advance >>> Thanks >>> M. Kamra >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 09:55:59 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sat, 13 Jul 2019 23:55:59 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Hello Matt, I revised my code and changed the way I create the rhs vector, previosly I was using vecCreateGhost just in case I need the ghost values, but for now I changed that to vecCreateMPI(.......) So maybe that was the cause of the scatter I am attaching with this email a new log output Also regarding how I fill my petsc matrix, In my code I fill a temp CSR format matrix becasue otherwise I would need "MatSetValue" to fill the petsc mat element by element which is not recommmeded in the petsc manual and probably very expensive due to function call overhead *So after I create my matrix in CSR format, I fill the PETSC mat A as follows* > for (i = 0; i < nMatRows; i++) { > cffset = CSR_iptr[i]; > row_index = row_gIndex[i]; > nj = Eqn_nj[i]; > MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, > CSR_vptr + offset, INSERT_VALUES); > } > *After That* > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > *I don't believe , I am doing anything special, if possible I would like to set the whole csr matrix at once in one command.* *I took a look at the code for MatSetValues, if I am understanding it correctly(hopefully) I think I could do it, maybe modify it or create a new routine entirely for this pupose.* *i.e. MatSetValuesFromCSR(.....)* *Or is there a particular reason why it has to be this way* I also tried ksp ex3 but I slightly tweaked it to add a logging stage around the assembly and MatSetValues and I am attaching the modified example here as well. Although in this example the matrix stash is not empty ( means off-processor values are being set ) but the timing values for roughly the same matrix size , the command I used is mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info Regards, Kamra On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley wrote: > On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Hello Matt, >> Attached is the dumped entire log output using -log_view and -info. >> > > In matrix construction, it looks like you have a mixture of load imbalance > (see the imbalance in the Begin events) > and lots of Scatter messages in your assembly. We turn off MatSetValues() > logging by default since it is usually > called many times, but you can explicitly turn it back on if you want. I > don't think that is the problem here. Its easy > to see from examples (say SNES ex5) that it is not the major time sink. > What is the Scatter doing? > > Thanks, > > Matt > > >> Thanks, >> Kamra >> >> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >> wrote: >> >>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello all, >>>> I have a few question regarding Petsc, >>>> >>> >>> Please send the entire output of a run with all the logging turned on, >>> using -log_view and -info. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Question 1: >>>> For the profiling , is it possible to only show the user defined log >>>> events in the breakdown of each stage in Log-view. >>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>> PetscLogEventExcludeClass(PC_CLASSID); >>>> which should "Deactivates event logging for a PETSc object class in >>>> every stage" according to the manual. >>>> however I still see them in the stage breakdown >>>> --- Event Stage 1: Matrix Construction >>>> >>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>> >>>> Also is possible to clear the logs so that I can write a separate >>>> profiling output file for each timestep ( since I am solving a transient >>>> problem and I want to know the change in performance as time goes by ) >>>> >>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> Question 2: >>>> Regarding MatSetValues >>>> Right now, I writing a finite volume code, due to algorithm requirement >>>> I have to write the matrix into local native format ( array of arrays) and >>>> then loop through rows and use MatSetValues to set the elements in "Mat A" >>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>> but it is very slow and it is killing my performance >>>> although the matrix was properly set using >>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>> PETSC_DETERMINE, >>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>> matsetvalues and all inserted values are local so no off-processor values >>>> So my question is it possible to set multiple rows at once hopefully >>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>> because it seems that row by row is expensive >>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>> data, as I mentioned all values are local and no off-processor values ( >>>> stash is 0 ) >>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>> mallocs. >>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>> space: 0 unneeded,743028 used >>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,742972 used >>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>> space: 0 unneeded,743093 used >>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,743036 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,742938 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>> space: 0 unneeded,743049 used >>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >>>> 0 unneeded,685 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >>>> 0 unneeded,649 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: >>>> 0 unneeded,1011 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: >>>> 0 unneeded,1137 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >>>> 0 unneeded,658 used >>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >>>> 0 unneeded,648 used >>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>> >>>> >>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> Question 3: >>>> If all matrix and vector inserted data are local, what part of the >>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>> more time than matrix builder >>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>> >>>> >>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>> six processes and it was NOT built using DM >>>> >>>> Finally the configure options are: >>>> >>>> Configure options: >>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>> --with-fc=mpif90 --download-metis --download-hypre >>>> >>>> Sorry for such long question and thanks in advance >>>> Thanks >>>> M. Kamra >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0] PetscInitialize(): PETSc successfully started: number of processors = 6 [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [1] PetscInitialize(): PETSc successfully started: number of processors = 6 [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [2] PetscInitialize(): PETSc successfully started: number of processors = 6 [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [3] PetscInitialize(): PETSc successfully started: number of processors = 6 [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [4] PetscInitialize(): PETSc successfully started: number of processors = 6 [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [5] PetscInitialize(): PETSc successfully started: number of processors = 6 [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 Processor: 0/6 Set Matrix Start [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. [4] MatSeqAIJCheckInode(): Found 186063 nodes out of 186063 rows. Not using Inode routines [5] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [0] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] MatSeqAIJCheckInode(): Found 186063 nodes out of 186063 rows. Not using Inode routines [1] MatSeqAIJCheckInode(): Found 186062 nodes out of 186062 rows. Not using Inode routines [2] MatSeqAIJCheckInode(): Found 186064 nodes out of 186064 rows. Not using Inode routines [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. Time to Setup (III): 0.0844824, count = 1 [1] PetscFinalize(): PetscFinalize() called [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscFinalize(): PetscFinalize() called [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscFinalize(): PetscFinalize() called [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscFinalize(): PetscFinalize() called [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscFinalize(): PetscFinalize() called [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscFinalize(): PetscFinalize() called [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ./OpenAqua300 on a release3 named mohamedkamra-DAIV-DQZ520 with 6 processors, by mohamedkamra Sat Jul 13 22:45:02 2019 Using Petsc Release Version 3.11.2, May, 18, 2019 Max Max/Min Avg Total Time (sec): 5.860e+00 1.000 5.860e+00 Objects: 5.300e+01 1.000 5.300e+01 Flop: 0.000e+00 0.000 0.000e+00 0.000e+00 Flop/sec: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Messages: 1.150e+02 2.500 6.900e+01 4.140e+02 MPI Message Lengths: 1.684e+05 1.755 1.713e+03 7.091e+05 MPI Reductions: 7.100e+01 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 5.7764e+00 98.6% 0.0000e+00 0.0% 3.780e+02 91.3% 1.825e+03 97.3% 5.600e+01 78.9% 1: Matrix Construction: 8.3482e-02 1.4% 0.0000e+00 0.0% 3.600e+01 8.7% 5.340e+02 2.7% 8.000e+00 11.3% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 9 1.0 7.4974e-0261.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecSet 17 1.0 4.9084e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAssemblyBegin 18 1.0 7.5064e-0256.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAssemblyEnd 18 1.0 4.4107e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 9 1.0 4.0770e-04 1.5 0.00e+00 0.0 1.6e+02 3.5e+03 0.0e+00 0 0 39 81 0 0 0 43 83 0 0 VecScatterEnd 9 1.0 6.4468e-04 6.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: Matrix Construction BuildTwoSidedF 2 1.0 2.2705e-02244.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 15 0 0 0 0 0 VecSet 1 1.0 3.5763e-06 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 1 1.0 2.2648e-02730.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 15 0 0 0 0 0 VecAssemblyEnd 1 1.0 4.5300e-06 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 1 1.0 1.0395e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 1 1.0 8.8344e-03 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 0 0 9 3 11 11 0100100100 0 AssembleMats 1 1.0 3.1597e-02 3.5 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 0 0 9 3 11 26 0100100100 0 myMatSetValues 1 1.0 2.2590e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 setNativeMat 1 1.0 4.7478e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 45 0 0 0 0 0 callScheme 1 1.0 7.2021e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0 --- Event Stage 2: Unknown --- Event Stage 3: Unknown ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 19 0 0 0. Index Set 12 12 25056 0. IS L to G Mapping 6 0 0 0. Vec Scatter 6 0 0 0. Matrix 3 0 0 0. Krylov Solver 1 0 0 0. Viewer 1 0 0 0. --- Event Stage 1: Matrix Construction Vector 2 1 1664 0. Index Set 2 2 4176 0. Vec Scatter 1 0 0 0. --- Event Stage 2: Unknown --- Event Stage 3: Unknown ======================================================================================================================== Average time to get PetscTime(): 7.62939e-07 Average time for MPI_Barrier(): 4.33922e-06 Average time for zero size MPI_Send(): 1.3113e-06 [4] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [1] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [2] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [3] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [5] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 #PETSc Option Table entries: -info -log_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre ----------------------------------------- Libraries compiled on 2019-07-03 09:57:26 on mohamedkamra-DAIV-DQZ520 Machine characteristics: Linux-4.18.0-25-generic-x86_64-with-Ubuntu-18.04-bionic Using PETSc directory: /home/mohamedkamra/Downloads/petsc-3.11.2 Using PETSc arch: release3 ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 -march=native -mtune=native Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -march=native -mtune=native ----------------------------------------- Using include paths: -I/home/mohamedkamra/Downloads/petsc-3.11.2/include -I/home/mohamedkamra/Downloads/petsc-3.11.2/release3/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -lpetsc -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lHYPRE -llapack -lblas -lmetis -lm -lX11 -lpthread -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl ----------------------------------------- [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374780 -------------- next part -------------- [0] PetscInitialize(): PETSc successfully started: number of processors = 6 [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [1] PetscInitialize(): PETSc successfully started: number of processors = 6 [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [2] PetscInitialize(): PETSc successfully started: number of processors = 6 [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [3] PetscInitialize(): PETSc successfully started: number of processors = 6 [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [4] PetscInitialize(): PETSc successfully started: number of processors = 6 [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [5] PetscInitialize(): PETSc successfully started: number of processors = 6 [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscInitialize(): Running on machine: mohamedkamra-DAIV-DQZ520 [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 268435455 [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [2] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [3] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [4] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [5] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 268435455 [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 268435455 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] MatAssemblyBegin_MPIAIJ(): Stash has 6660 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 6672 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 6664 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 6660 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 6672 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 6672 entries, uses 0 mallocs. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 7006 unneeded,1495994 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 7006 unneeded,1495994 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 7006 unneeded,1495994 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 7006 unneeded,1495994 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 7002 unneeded,1495998 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 167001; storage space: 7002 unneeded,1496007 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167001) < 0.6. Do not use CompressedRow routines. [4] MatSeqAIJCheckInode(): Found 167000 nodes out of 167000 rows. Not using Inode routines [2] MatSeqAIJCheckInode(): Found 167000 nodes out of 167000 rows. Not using Inode routines [3] MatSeqAIJCheckInode(): Found 167000 nodes out of 167000 rows. Not using Inode routines [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] MatSeqAIJCheckInode(): Found 167001 nodes out of 167001 rows. Not using Inode routines [1] MatSeqAIJCheckInode(): Found 167000 nodes out of 167000 rows. Not using Inode routines [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] MatSeqAIJCheckInode(): Found 167000 nodes out of 167000 rows. Not using Inode routines [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreate_MPI1(): Using MPI1 for vector scatter [0] VecScatterCreateCommon_PtoS_MPI1(): Using blocksize 1 scatter [0] VecScatterCreate_PtoS(): General case: MPI to Seq [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 1329996 unneeded,6004 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 164996)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 1329996 unneeded,6004 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 164996)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 1329996 unneeded,6004 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 164996)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 1002; storage space: 1332998 unneeded,3002 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 165998)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 1329996 unneeded,6004 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 164996)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 1002; storage space: 1333006 unneeded,3002 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 165999)/(num_localrows 167001) > 0.6. Use CompressedRow routines. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] VecAssemblyBegin_MPI_BTS(): Stash has 834 entries, uses 5 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 167001; storage space: 6652 unneeded,1489355 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167001) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 1662 unneeded,1494332 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 1662 unneeded,1494332 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 1662 unneeded,1494332 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 1002; storage space: 4 unneeded,2998 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 166001)/(num_localrows 167001) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 8 unneeded,5996 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 8 unneeded,5996 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 8 unneeded,5996 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 1662 unneeded,1494332 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 6652 unneeded,1489346 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 8 unneeded,5996 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 1002; storage space: 4 unneeded,2998 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 166000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 167001; storage space: 0 unneeded,1489355 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167001) < 0.6. Do not use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 0 unneeded,1494332 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 0 unneeded,1494332 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 0 unneeded,1494332 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 0 unneeded,1494332 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 167001 X 1002; storage space: 0 unneeded,2998 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 0 unneeded,5996 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 167000; storage space: 0 unneeded,1489346 used [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 166001)/(num_localrows 167001) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 0 unneeded,5996 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 167000) < 0.6. Do not use CompressedRow routines. [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 0 unneeded,5996 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 2004; storage space: 0 unneeded,5996 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 165000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 167000 X 1002; storage space: 0 unneeded,2998 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 166000)/(num_localrows 167000) > 0.6. Use CompressedRow routines. [0] VecAssemblyBegin_MPI_BTS(): Stash has 2668 entries, uses 5 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 2668 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PCSetUp(): Setting up PC for first time [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PCSetUp(): Setting up PC for first time [0] PCSetUp(): Setting up PC for first time [2] PCSetUp(): Setting up PC for first time [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PCSetUp(): Setting up PC for first time [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PCSetUp(): Setting up PC for first time [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PCSetUp(): Setting up PC for first time [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] KSPConvergedDefault(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [5] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [1] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [2] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [4] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [3] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] KSPConvergedDefault(): Linear solver has converged. Residual norm 5.834635436978e-04 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 5.841577009637e+01 at iteration 955 [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. Norm of error 0.00480768 Iterations 955 [5] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374777 [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374777 [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374777 [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [0] PetscFinalize(): PetscFinalize() called [5] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [5] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [5] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [2] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [5] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [5] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [5] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [5] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [5] PetscFinalize(): PetscFinalize() called [2] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [2] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [2] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [2] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [2] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [4] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [4] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [4] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [4] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [2] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [4] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [4] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [4] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [4] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [4] PetscFinalize(): PetscFinalize() called [2] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [2] PetscFinalize(): PetscFinalize() called [3] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [1] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [3] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [3] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [3] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [1] Petsc_DelComm_Outer(): User MPI_Comm 1140850689 is being freed after removing reference from inner PETSc comm to this outer comm [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [1] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [3] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [1] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [3] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [3] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [3] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [1] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [3] PetscFinalize(): PetscFinalize() called [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [1] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [1] PetscFinalize(): PetscFinalize() called [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 268435455 [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [2] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [3] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [4] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [5] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 268435455 [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [4] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [5] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [0] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [1] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) ./OpenAqua300 on a release3 named mohamedkamra-DAIV-DQZ520 with 6 processors, by mohamedkamra Sat Jul 13 23:38:54 2019 Using Petsc Release Version 3.11.2, May, 18, 2019 [2] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) [3] PetscGetHostName(): Rejecting domainname, likely is NIS mohamedkamra-DAIV-DQZ520.(none) Max Max/Min Avg Total Time (sec): 2.362e+01 1.000 2.362e+01 Objects: 5.800e+01 1.000 5.800e+01 Flop: 1.627e+10 1.002 1.626e+10 9.759e+10 Flop/sec: 6.889e+08 1.002 6.886e+08 4.131e+09 MPI Messages: 2.024e+03 1.964 1.692e+03 1.016e+04 MPI Message Lengths: 1.603e+07 1.974 7.911e+03 8.034e+07 MPI Reductions: 1.965e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 2.3531e+01 99.6% 9.7586e+10 100.0% 1.012e+04 99.7% 7.887e+03 99.4% 1.950e+03 99.2% 1: CreateMatrix: 5.2341e-02 0.2% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 3.7130e-02 0.2% 0.0000e+00 0.0% 3.500e+01 0.3% 1.487e+04 0.6% 8.000e+00 0.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 1 1.0 3.0565e-04 5.9 0.00e+00 0.0 1.5e+01 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 BuildTwoSidedF 5 1.0 5.7135e-03 6.0 0.00e+00 0.0 1.8e+02 3.1e+03 0.0e+00 0 0 2 1 0 0 0 2 1 0 0 MatMult 987 1.0 5.3681e+00 1.0 2.80e+09 1.0 9.9e+03 8.0e+03 0.0e+00 22 17 97 98 0 22 17 98 99 0 3120 MatSolve 988 1.0 5.6464e+00 1.0 2.79e+09 1.0 0.0e+00 0.0e+00 0.0e+00 24 17 0 0 0 24 17 0 0 0 2959 MatLUFactorNum 1 1.0 1.0801e-02 1.1 6.13e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3399 MatILUFactorSym 1 1.0 1.4261e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 1 1.0 1.9233e-0322.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 1 1.0 4.6010e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 6.9141e-06 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.5485e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMDot 955 1.0 5.4459e+00 1.1 4.92e+09 1.0 0.0e+00 0.0e+00 9.6e+02 22 30 0 0 49 22 30 0 0 49 5424 VecNorm 989 1.0 7.8345e-01 1.3 3.30e+08 1.0 0.0e+00 0.0e+00 9.9e+02 3 2 0 0 50 3 2 0 0 51 2530 VecScale 987 1.0 2.8213e-01 1.0 1.65e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 3505 VecCopy 32 1.0 1.2547e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1022 1.0 3.6945e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecAXPY 65 1.0 4.8391e-02 1.2 2.17e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2692 VecMAXPY 987 1.0 5.8750e+00 1.0 5.24e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 32 0 0 0 25 32 0 0 0 5354 VecAssemblyBegin 4 1.0 4.4274e-03 2.7 0.00e+00 0.0 1.8e+02 3.1e+03 0.0e+00 0 0 2 1 0 0 0 2 1 0 0 VecAssemblyEnd 4 1.0 2.1100e-04 4.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 987 1.0 1.6651e-02 1.5 0.00e+00 0.0 9.9e+03 8.0e+03 0.0e+00 0 0 97 98 0 0 0 98 99 0 0 VecScatterEnd 987 1.0 2.6902e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 987 1.0 1.0784e+00 1.2 4.94e+08 1.0 0.0e+00 0.0e+00 9.9e+02 4 3 0 0 50 4 3 0 0 51 2751 SFSetGraph 1 1.0 1.6689e-06 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 1 1.0 4.0174e-04 2.0 0.00e+00 0.0 4.5e+01 1.8e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceBegin 1 1.0 2.6464e-05 1.6 0.00e+00 0.0 3.0e+01 2.7e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceEnd 1 1.0 2.9325e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 2 1.0 2.9552e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1 1.0 2.3490e+01 1.0 1.63e+10 1.0 9.9e+03 8.0e+03 1.9e+03 99100 97 98 99 100100 98 99100 4154 KSPGMRESOrthog 955 1.0 1.0968e+01 1.0 9.85e+09 1.0 0.0e+00 0.0e+00 9.6e+02 46 61 0 0 49 46 61 0 0 49 5386 PCSetUp 2 1.0 2.5425e-02 1.2 6.13e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1444 PCSetUpOnBlocks 1 1.0 2.5289e-02 1.2 6.13e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1452 PCApply 988 1.0 6.0802e+00 1.0 2.79e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 17 0 0 0 25 17 0 0 0 2748 --- Event Stage 1: CreateMatrix --- Event Stage 2: AssembleMatrix BuildTwoSidedF 1 1.0 2.3713e-0252.5 0.00e+00 0.0 1.5e+01 3.2e+04 0.0e+00 0 0 0 1 0 34 0 43 92 0 0 MatAssemblyBegin 1 1.0 2.4225e-0220.5 0.00e+00 0.0 1.5e+01 3.2e+04 0.0e+00 0 0 0 1 0 35 0 43 92 0 0 MatAssemblyEnd 1 1.0 2.4207e-02 1.0 0.00e+00 0.0 2.0e+01 2.0e+03 8.0e+00 0 0 0 0 0 65 0 57 8100 0 VecSet 1 1.0 5.2452e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 4 4 61309196 0. Vector 40 41 50844160 0. Index Set 3 3 2006388 0. Vec Scatter 0 1 1392 0. Star Forest Graph 1 1 888 0. Krylov Solver 2 2 20040 0. Preconditioner 2 2 1912 0. Viewer 1 0 0 0. --- Event Stage 1: CreateMatrix --- Event Stage 2: AssembleMatrix Vector 2 1 1664 0. Index Set 2 2 5592 0. Vec Scatter 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 4.76837e-08 Average time for zero size MPI_Send(): 0. #PETSc Option Table entries: -info -log_view -m 1000 #End of PETSc Option Table entries Compiled without FORTRAN kernels [1] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [2] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [2] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [2] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [2] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [2] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [3] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [3] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [3] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [3] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [3] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [4] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [4] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [4] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [4] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [4] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 [5] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374782 [5] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [5] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [5] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [5] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre ----------------------------------------- Libraries compiled on 2019-07-03 09:57:26 on mohamedkamra-DAIV-DQZ520 Machine characteristics: Linux-4.18.0-25-generic-x86_64-with-Ubuntu-18.04-bionic Using PETSc directory: /home/mohamedkamra/Downloads/petsc-3.11.2 Using PETSc arch: release3 ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 -march=native -mtune=native Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -march=native -mtune=native ----------------------------------------- Using include paths: -I/home/mohamedkamra/Downloads/petsc-3.11.2/include -I/home/mohamedkamra/Downloads/petsc-3.11.2/release3/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -lpetsc -Wl,-rpath,/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -L/home/mohamedkamra/Downloads/petsc-3.11.2/release3/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lHYPRE -llapack -lblas -lmetis -lm -lX11 -lpthread -lstdc++ -ldl -lmpichfort -lmpich -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl ----------------------------------------- [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374780 [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374780 [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374780 [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374780 [1] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374782 [1] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374782 [1] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374782 -------------- next part -------------- A non-text attachment was scrubbed... Name: mod_ksp_ex3.c Type: text/x-csrc Size: 8625 bytes Desc: not available URL: From knepley at gmail.com Sat Jul 13 10:41:44 2019 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Jul 2019 10:41:44 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa wrote: > Hello Matt, > > I revised my code and changed the way I create the rhs vector, > previosly I was using vecCreateGhost just in case I need the ghost values, > but for now I changed that to > vecCreateMPI(.......) > So maybe that was the cause of the scatter > I am attaching with this email a new log output > Okay, the times are now very small. How does it scale up? Thanks, Matt > Also regarding how I fill my petsc matrix, > In my code I fill a temp CSR format matrix becasue otherwise I would need > "MatSetValue" to fill the petsc mat element by element > which is not recommmeded in the petsc manual and probably very expensive > due to function call overhead > *So after I create my matrix in CSR format, I fill the PETSC mat A as > follows* > >> for (i = 0; i < nMatRows; i++) { >> cffset = CSR_iptr[i]; >> row_index = row_gIndex[i]; >> nj = Eqn_nj[i]; >> MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >> CSR_vptr + offset, INSERT_VALUES); >> } >> > *After That* > >> VecAssemblyBegin(RHS); >> VecAssemblyEnd(RHS); >> >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> > > *I don't believe , I am doing anything special, if possible I would like > to set the whole csr matrix at once in one command.* > *I took a look at the code for MatSetValues, if I am understanding it > correctly(hopefully) I think I could do it, maybe modify it or create a new > routine entirely for this pupose.* > > *i.e. MatSetValuesFromCSR(.....)* > *Or is there a particular reason why it has to be this way* > > I also tried ksp ex3 but I slightly tweaked it to add a logging stage > around the assembly and MatSetValues and I am attaching the modified > example here as well. > Although in this example the matrix stash is not empty ( means > off-processor values are being set ) but the timing values for roughly the > same matrix size , the command I used is > mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info > > > Regards, > Kamra > > On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley wrote: > >> On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> Hello Matt, >>> Attached is the dumped entire log output using -log_view and -info. >>> >> >> In matrix construction, it looks like you have a mixture of load >> imbalance (see the imbalance in the Begin events) >> and lots of Scatter messages in your assembly. We turn off MatSetValues() >> logging by default since it is usually >> called many times, but you can explicitly turn it back on if you want. I >> don't think that is the problem here. Its easy >> to see from examples (say SNES ex5) that it is not the major time sink. >> What is the Scatter doing? >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Kamra >>> >>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>> wrote: >>> >>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Hello all, >>>>> I have a few question regarding Petsc, >>>>> >>>> >>>> Please send the entire output of a run with all the logging turned on, >>>> using -log_view and -info. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Question 1: >>>>> For the profiling , is it possible to only show the user defined log >>>>> events in the breakdown of each stage in Log-view. >>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>> which should "Deactivates event logging for a PETSc object class in >>>>> every stage" according to the manual. >>>>> however I still see them in the stage breakdown >>>>> --- Event Stage 1: Matrix Construction >>>>> >>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>> >>>>> Also is possible to clear the logs so that I can write a separate >>>>> profiling output file for each timestep ( since I am solving a transient >>>>> problem and I want to know the change in performance as time goes by ) >>>>> >>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> Question 2: >>>>> Regarding MatSetValues >>>>> Right now, I writing a finite volume code, due to algorithm >>>>> requirement I have to write the matrix into local native format ( array of >>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>> in "Mat A" >>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>> but it is very slow and it is killing my performance >>>>> although the matrix was properly set using >>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>> PETSC_DETERMINE, >>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>> matsetvalues and all inserted values are local so no off-processor values >>>>> So my question is it possible to set multiple rows at once hopefully >>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>> because it seems that row by row is expensive >>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>> data, as I mentioned all values are local and no off-processor values ( >>>>> stash is 0 ) >>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>> mallocs. >>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>> space: 0 unneeded,743028 used >>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,742972 used >>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>> space: 0 unneeded,743093 used >>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,743036 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,742938 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>> space: 0 unneeded,743049 used >>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >>>>> 0 unneeded,685 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >>>>> 0 unneeded,649 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>> space: 0 unneeded,1011 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>> space: 0 unneeded,1137 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >>>>> 0 unneeded,658 used >>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >>>>> 0 unneeded,648 used >>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>> >>>>> >>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> Question 3: >>>>> If all matrix and vector inserted data are local, what part of the >>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>> more time than matrix builder >>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>> >>>>> >>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>> six processes and it was NOT built using DM >>>>> >>>>> Finally the configure options are: >>>>> >>>>> Configure options: >>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>> >>>>> Sorry for such long question and thanks in advance >>>>> Thanks >>>>> M. Kamra >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 11:20:11 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 01:20:11 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: I am sorry but I don?t see what you mean by small times Although mat assembly is relatively smaller The cost of mat set values is still significant The same can be said for vec assembly Combined vec/mat assembly and matsetvalues constitute about 50% of the total cost of matrix construction So is this problem of my matrix setup/ preallocation Or is this a hardware issue, for whatever reason the copy is overly slow The code was run on a single node Or is this function call overhead since matsetvalues is being called 1M times inside the for loop ( 170k times in each process) Thanks, Kamra On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley wrote: > On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Hello Matt, >> >> I revised my code and changed the way I create the rhs vector, >> previosly I was using vecCreateGhost just in case I need the ghost >> values, but for now I changed that to >> vecCreateMPI(.......) >> So maybe that was the cause of the scatter >> I am attaching with this email a new log output >> > > Okay, the times are now very small. How does it scale up? > > Thanks, > > Matt > > >> Also regarding how I fill my petsc matrix, >> In my code I fill a temp CSR format matrix becasue otherwise I would need >> "MatSetValue" to fill the petsc mat element by element >> which is not recommmeded in the petsc manual and probably very expensive >> due to function call overhead >> *So after I create my matrix in CSR format, I fill the PETSC mat A as >> follows* >> >>> for (i = 0; i < nMatRows; i++) { >>> cffset = CSR_iptr[i]; >>> row_index = row_gIndex[i]; >>> nj = Eqn_nj[i]; >>> MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >>> CSR_vptr + offset, INSERT_VALUES); >>> } >>> >> *After That* >> >>> VecAssemblyBegin(RHS); >>> VecAssemblyEnd(RHS); >>> >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> >> >> *I don't believe , I am doing anything special, if possible I would like >> to set the whole csr matrix at once in one command.* >> *I took a look at the code for MatSetValues, if I am understanding it >> correctly(hopefully) I think I could do it, maybe modify it or create a new >> routine entirely for this pupose.* >> >> *i.e. MatSetValuesFromCSR(.....)* >> *Or is there a particular reason why it has to be this way* >> >> I also tried ksp ex3 but I slightly tweaked it to add a logging stage >> around the assembly and MatSetValues and I am attaching the modified >> example here as well. >> Although in this example the matrix stash is not empty ( means >> off-processor values are being set ) but the timing values for roughly the >> same matrix size , the command I used is >> mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info >> >> >> Regards, >> Kamra >> >> On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley >> wrote: >> >>> On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> >>>> Hello Matt, >>>> Attached is the dumped entire log output using -log_view and -info. >>>> >>> >>> In matrix construction, it looks like you have a mixture of load >>> imbalance (see the imbalance in the Begin events) >>> and lots of Scatter messages in your assembly. We turn off >>> MatSetValues() logging by default since it is usually >>> called many times, but you can explicitly turn it back on if you want. I >>> don't think that is the problem here. Its easy >>> to see from examples (say SNES ex5) that it is not the major time sink. >>> What is the Scatter doing? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Kamra >>>> >>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>> petsc-users at mcs.anl.gov> wrote: >>>>> >>>>>> Hello all, >>>>>> I have a few question regarding Petsc, >>>>>> >>>>> >>>>> Please send the entire output of a run with all the logging turned on, >>>>> using -log_view and -info. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Question 1: >>>>>> For the profiling , is it possible to only show the user defined log >>>>>> events in the breakdown of each stage in Log-view. >>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>> every stage" according to the manual. >>>>>> however I still see them in the stage breakdown >>>>>> --- Event Stage 1: Matrix Construction >>>>>> >>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>> >>>>>> Also is possible to clear the logs so that I can write a separate >>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>> problem and I want to know the change in performance as time goes by ) >>>>>> >>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>> Question 2: >>>>>> Regarding MatSetValues >>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>> requirement I have to write the matrix into local native format ( array of >>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>> in "Mat A" >>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>> but it is very slow and it is killing my performance >>>>>> although the matrix was properly set using >>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>> PETSC_DETERMINE, >>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>> So my question is it possible to set multiple rows at once hopefully >>>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>>> because it seems that row by row is expensive >>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>> stash is 0 ) >>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>> mallocs. >>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>> space: 0 unneeded,743028 used >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,742972 used >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>> space: 0 unneeded,743093 used >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,743036 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,742938 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>> space: 0 unneeded,743049 used >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>> space: 0 unneeded,685 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>> space: 0 unneeded,649 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>> space: 0 unneeded,1011 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>> space: 0 unneeded,1137 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>> space: 0 unneeded,658 used >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>> space: 0 unneeded,648 used >>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>> >>>>>> >>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>> Question 3: >>>>>> If all matrix and vector inserted data are local, what part of the >>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>> more time than matrix builder >>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>> >>>>>> >>>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>>> six processes and it was NOT built using DM >>>>>> >>>>>> Finally the configure options are: >>>>>> >>>>>> Configure options: >>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>> >>>>>> Sorry for such long question and thanks in advance >>>>>> Thanks >>>>>> M. Kamra >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jul 13 11:50:54 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 13 Jul 2019 16:50:54 +0000 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: <3BFD3707-6543-467A-8A1C-17A27B1A306C@anl.gov> How are you generating entries in your matrix? Finite differences, finite element, finite volume, something else? If you are using finite differences you generally generate an entire row at a time and call MatSetValues() once per row. With finite elements you generates an element at a time and ADD_VALUES for a block of rows and columns. I don't know why generating directly in CSR format would be faster than than calling MatSetValues() once per row but anyways if you have the matrix in CSR format you can use MatCreateMPIAIJWithArrays() (and in the master branch of the repository) MatUpdateMPIAIJWithArrays(). to build the matrix the first time, and then "refill" it with numerical values each new time. There are a few other optimizations related to matrix insertion in the master branch you might also benefit from. Generally for problems with multiple "times" or "linear solve steps" we use two stages, the first to track the initial set up and first time step and the other to capture all the other steps (since the extra overhead is only in the first step.) You could make a new stage for each time step but I don't think that is needed. After you have this going send us the new log summary. Barry > On Jul 13, 2019, at 11:20 AM, Mohammed Mostafa via petsc-users wrote: > > I am sorry but I don?t see what you mean by small times > Although mat assembly is relatively smaller > The cost of mat set values is still significant > The same can be said for vec assembly > Combined vec/mat assembly and matsetvalues constitute about 50% of the total cost of matrix construction > > So is this problem of my matrix setup/ preallocation > > Or is this a hardware issue, for whatever reason the copy is overly slow > The code was run on a single node > > Or is this function call overhead since matsetvalues is being called 1M times inside the for loop ( 170k times in each process) > > Thanks, Kamra > > On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley wrote: > On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa wrote: > Hello Matt, > > I revised my code and changed the way I create the rhs vector, > previosly I was using vecCreateGhost just in case I need the ghost values, but for now I changed that to > vecCreateMPI(.......) > So maybe that was the cause of the scatter > I am attaching with this email a new log output > > Okay, the times are now very small. How does it scale up? > > Thanks, > > Matt > > Also regarding how I fill my petsc matrix, > In my code I fill a temp CSR format matrix becasue otherwise I would need "MatSetValue" to fill the petsc mat element by element > which is not recommmeded in the petsc manual and probably very expensive due to function call overhead > So after I create my matrix in CSR format, I fill the PETSC mat A as follows > for (i = 0; i < nMatRows; i++) { > cffset = CSR_iptr[i]; > row_index = row_gIndex[i]; > nj = Eqn_nj[i]; > MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, CSR_vptr + offset, INSERT_VALUES); > } > After That > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > I don't believe , I am doing anything special, if possible I would like to set the whole csr matrix at once in one command. > I took a look at the code for MatSetValues, if I am understanding it correctly(hopefully) I think I could do it, maybe modify it or create a new routine entirely for this pupose. > i.e. MatSetValuesFromCSR(.....) > Or is there a particular reason why it has to be this way > > I also tried ksp ex3 but I slightly tweaked it to add a logging stage around the assembly and MatSetValues and I am attaching the modified example here as well. > Although in this example the matrix stash is not empty ( means off-processor values are being set ) but the timing values for roughly the same matrix size , the command I used is > mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info > > > Regards, > Kamra > > On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley wrote: > On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa wrote: > Hello Matt, > Attached is the dumped entire log output using -log_view and -info. > > In matrix construction, it looks like you have a mixture of load imbalance (see the imbalance in the Begin events) > and lots of Scatter messages in your assembly. We turn off MatSetValues() logging by default since it is usually > called many times, but you can explicitly turn it back on if you want. I don't think that is the problem here. Its easy > to see from examples (say SNES ex5) that it is not the major time sink. What is the Scatter doing? > > Thanks, > > Matt > > Thanks, > Kamra > > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley wrote: > On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users wrote: > Hello all, > I have a few question regarding Petsc, > > Please send the entire output of a run with all the logging turned on, using -log_view and -info. > > Thanks, > > Matt > > Question 1: > For the profiling , is it possible to only show the user defined log events in the breakdown of each stage in Log-view. > I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, > PetscLogEventExcludeClass(MAT_CLASSID); > PetscLogEventExcludeClass(VEC_CLASSID); > PetscLogEventExcludeClass(KSP_CLASSID); > PetscLogEventExcludeClass(PC_CLASSID); > which should "Deactivates event logging for a PETSc object class in every stage" according to the manual. > however I still see them in the stage breakdown > --- Event Stage 1: Matrix Construction > > BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 > VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 > AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT > myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT > setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT > setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT > callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT > > Also is possible to clear the logs so that I can write a separate profiling output file for each timestep ( since I am solving a transient problem and I want to know the change in performance as time goes by ) > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Question 2: > Regarding MatSetValues > Right now, I writing a finite volume code, due to algorithm requirement I have to write the matrix into local native format ( array of arrays) and then loop through rows and use MatSetValues to set the elements in "Mat A" > MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); > but it is very slow and it is killing my performance > although the matrix was properly set using > MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, PETSC_DETERMINE, > PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); > with d_nnz,and o_nnz properly assigned so no mallocs occur during matsetvalues and all inserted values are local so no off-processor values > So my question is it possible to set multiple rows at once hopefully all, I checked the manual and MatSetValues can only set dense matrix block because it seems that row by row is expensive > Or perhaps is it possible to copy all rows to the underlying matrix data, as I mentioned all values are local and no off-processor values ( stash is 0 ) > [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. > [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Question 3: > If all matrix and vector inserted data are local, what part of the vec/mat assembly consumes time because matsetvalues and matassembly consume more time than matrix builder > Also this is not just for the first time MAT_FINAL_ASSEMBLY > > > For context the matrix in the above is nearly 1Mx1M partitioned over six processes and it was NOT built using DM > > Finally the configure options are: > > Configure options: > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre > > Sorry for such long question and thanks in advance > Thanks > M. Kamra > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Sat Jul 13 11:51:55 2019 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Jul 2019 11:51:55 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Sat, Jul 13, 2019 at 11:20 AM Mohammed Mostafa wrote: > I am sorry but I don?t see what you mean by small times > Although mat assembly is relatively smaller > The cost of mat set values is still significant > The same can be said for vec assembly > Combined vec/mat assembly and matsetvalues constitute about 50% of the > total cost of matrix construction > This is why I asked you about scaling, since it is difficult to disentangle overheads from scalable work at 10^-2 seconds. Second, if you look at the times for the PETSc example you ran, setvalues and assembly is clearly not 50% of the time for construction. I cannot see your code, so we do not know exactly what the custom events are timing. I would think about incrementally changing the example until you get what you want. Thanks, Matt > So is this problem of my matrix setup/ preallocation > > Or is this a hardware issue, for whatever reason the copy is overly slow > The code was run on a single node > > Or is this function call overhead since matsetvalues is being called 1M > times inside the for loop ( 170k times in each process) > > Thanks, Kamra > > On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley > wrote: > >> On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> Hello Matt, >>> >>> I revised my code and changed the way I create the rhs vector, >>> previosly I was using vecCreateGhost just in case I need the ghost >>> values, but for now I changed that to >>> vecCreateMPI(.......) >>> So maybe that was the cause of the scatter >>> I am attaching with this email a new log output >>> >> >> Okay, the times are now very small. How does it scale up? >> >> Thanks, >> >> Matt >> >> >>> Also regarding how I fill my petsc matrix, >>> In my code I fill a temp CSR format matrix becasue otherwise I would >>> need "MatSetValue" to fill the petsc mat element by element >>> which is not recommmeded in the petsc manual and probably very expensive >>> due to function call overhead >>> *So after I create my matrix in CSR format, I fill the PETSC mat A as >>> follows* >>> >>>> for (i = 0; i < nMatRows; i++) { >>>> cffset = CSR_iptr[i]; >>>> row_index = row_gIndex[i]; >>>> nj = Eqn_nj[i]; >>>> MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >>>> CSR_vptr + offset, INSERT_VALUES); >>>> } >>>> >>> *After That* >>> >>>> VecAssemblyBegin(RHS); >>>> VecAssemblyEnd(RHS); >>>> >>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>> >>> >>> *I don't believe , I am doing anything special, if possible I would like >>> to set the whole csr matrix at once in one command.* >>> *I took a look at the code for MatSetValues, if I am understanding it >>> correctly(hopefully) I think I could do it, maybe modify it or create a new >>> routine entirely for this pupose.* >>> >>> *i.e. MatSetValuesFromCSR(.....)* >>> *Or is there a particular reason why it has to be this way* >>> >>> I also tried ksp ex3 but I slightly tweaked it to add a logging stage >>> around the assembly and MatSetValues and I am attaching the modified >>> example here as well. >>> Although in this example the matrix stash is not empty ( means >>> off-processor values are being set ) but the timing values for roughly the >>> same matrix size , the command I used is >>> mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info >>> >>> >>> Regards, >>> Kamra >>> >>> On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley >>> wrote: >>> >>>> On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >>>> mo7ammedmostafa at gmail.com> wrote: >>>> >>>>> Hello Matt, >>>>> Attached is the dumped entire log output using -log_view and -info. >>>>> >>>> >>>> In matrix construction, it looks like you have a mixture of load >>>> imbalance (see the imbalance in the Begin events) >>>> and lots of Scatter messages in your assembly. We turn off >>>> MatSetValues() logging by default since it is usually >>>> called many times, but you can explicitly turn it back on if you want. >>>> I don't think that is the problem here. Its easy >>>> to see from examples (say SNES ex5) that it is not the major time sink. >>>> What is the Scatter doing? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Kamra >>>>> >>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>> >>>>>>> Hello all, >>>>>>> I have a few question regarding Petsc, >>>>>>> >>>>>> >>>>>> Please send the entire output of a run with all the logging turned >>>>>> on, using -log_view and -info. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Question 1: >>>>>>> For the profiling , is it possible to only show the user defined log >>>>>>> events in the breakdown of each stage in Log-view. >>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>>> every stage" according to the manual. >>>>>>> however I still see them in the stage breakdown >>>>>>> --- Event Stage 1: Matrix Construction >>>>>>> >>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>> >>>>>>> Also is possible to clear the logs so that I can write a separate >>>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>>> problem and I want to know the change in performance as time goes by ) >>>>>>> >>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>> Question 2: >>>>>>> Regarding MatSetValues >>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>> in "Mat A" >>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>> but it is very slow and it is killing my performance >>>>>>> although the matrix was properly set using >>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>>> PETSC_DETERMINE, >>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>>> So my question is it possible to set multiple rows at once hopefully >>>>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>>>> because it seems that row by row is expensive >>>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>>> stash is 0 ) >>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>> mallocs. >>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>>> space: 0 unneeded,743028 used >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,742972 used >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>> space: 0 unneeded,743093 used >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,743036 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,742938 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>> space: 0 unneeded,743049 used >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>> space: 0 unneeded,685 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>> space: 0 unneeded,649 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>> space: 0 unneeded,1011 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>> space: 0 unneeded,1137 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>> space: 0 unneeded,658 used >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>> space: 0 unneeded,648 used >>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>> Question 3: >>>>>>> If all matrix and vector inserted data are local, what part of the >>>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>>> more time than matrix builder >>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>> >>>>>>> >>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>>>> six processes and it was NOT built using DM >>>>>>> >>>>>>> Finally the configure options are: >>>>>>> >>>>>>> Configure options: >>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>>> >>>>>>> Sorry for such long question and thanks in advance >>>>>>> Thanks >>>>>>> M. Kamra >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 12:31:41 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 02:31:41 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Apply the scaling test to my code is such a hassle since I need to generate a family of meshes of different sizes so for now I applied it the ksp example code with m = 100,200,400,600,900,1350 I clipped the stage summary section for different m, Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total m=100 1: CreateMatrix: 4.5347e-04 2.5% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 1.2514e-03 6.8% 0.0000e+00 0.0% 3.500e+01 3.7% 1.499e+03 7.8% 8.000e+00 5.2% m=200 1: CreateMatrix: 2.1127e-03 3.2% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 2.3036e-03 3.4% 0.0000e+00 0.0% 3.500e+01 2.4% 2.985e+03 4.8% 8.000e+00 3.1% m=400 1: CreateMatrix: 8.8379e-03 0.9% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 5.1055e-03 0.5% 0.0000e+00 0.0% 3.500e+01 1.1% 5.956e+03 2.1% 8.000e+00 1.3% m=600 1: CreateMatrix: 1.2745e-02 0.3% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 1.4184e-02 0.3% 0.0000e+00 0.0% 3.500e+01 0.6% 8.923e+03 1.2% 8.000e+00 0.8% m=900 1: CreateMatrix: 4.1014e-02 0.2% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 2.6641e-02 0.2% 0.0000e+00 0.0% 3.500e+01 0.4% 1.338e+04 0.8% 8.000e+00 0.5% m=1350 1: CreateMatrix: 9.3838e-02 0.2% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2: AssembleMatrix: 5.2478e-02 0.1% 0.0000e+00 0.0% 3.500e+01 0.4% 2.007e+04 0.7% 8.000e+00 0.5% when summarized it becomes as follows m timings_CreateMatrix timings_AssembleMatrix m_ratio timing_ratio_setValue timing_ratio_assemble 100 4.53E-04 1.25E-03 1 1 1 200 2.11E-03 2.30E-03 2 4.65896310671048 1.84081828352245 400 8.84E-03 5.11E-03 2 4.18322525678042 2.21631359611044 600 1.27E-02 1.42E-02 1.5 1.44208465812014 2.77818039369308 900 4.10E-02 2.66E-02 1.5 3.21804629266379 1.87824309080654 1350 9.38E-02 5.25E-02 1.5 2.28795045594187 1.96982095266694 On Sun, Jul 14, 2019 at 1:52 AM Matthew Knepley wrote: > On Sat, Jul 13, 2019 at 11:20 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> I am sorry but I don?t see what you mean by small times >> Although mat assembly is relatively smaller >> The cost of mat set values is still significant >> The same can be said for vec assembly >> Combined vec/mat assembly and matsetvalues constitute about 50% of the >> total cost of matrix construction >> > > This is why I asked you about scaling, since it is difficult to > disentangle overheads from scalable work at 10^-2 seconds. > > Second, if you look at the times for the PETSc example you ran, setvalues > and assembly is clearly not 50% of the > time for construction. I cannot see your code, so we do not know exactly > what the custom events are timing. I > would think about incrementally changing the example until you get what > you want. > > Thanks, > > Matt > > >> So is this problem of my matrix setup/ preallocation >> >> Or is this a hardware issue, for whatever reason the copy is overly slow >> The code was run on a single node >> >> Or is this function call overhead since matsetvalues is being called 1M >> times inside the for loop ( 170k times in each process) >> >> Thanks, Kamra >> >> On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley >> wrote: >> >>> On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> >>>> Hello Matt, >>>> >>>> I revised my code and changed the way I create the rhs vector, >>>> previosly I was using vecCreateGhost just in case I need the ghost >>>> values, but for now I changed that to >>>> vecCreateMPI(.......) >>>> So maybe that was the cause of the scatter >>>> I am attaching with this email a new log output >>>> >>> >>> Okay, the times are now very small. How does it scale up? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Also regarding how I fill my petsc matrix, >>>> In my code I fill a temp CSR format matrix becasue otherwise I would >>>> need "MatSetValue" to fill the petsc mat element by element >>>> which is not recommmeded in the petsc manual and probably very >>>> expensive due to function call overhead >>>> *So after I create my matrix in CSR format, I fill the PETSC mat A as >>>> follows* >>>> >>>>> for (i = 0; i < nMatRows; i++) { >>>>> cffset = CSR_iptr[i]; >>>>> row_index = row_gIndex[i]; >>>>> nj = Eqn_nj[i]; >>>>> MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >>>>> CSR_vptr + offset, INSERT_VALUES); >>>>> } >>>>> >>>> *After That* >>>> >>>>> VecAssemblyBegin(RHS); >>>>> VecAssemblyEnd(RHS); >>>>> >>>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>>> >>>> >>>> *I don't believe , I am doing anything special, if possible I would >>>> like to set the whole csr matrix at once in one command.* >>>> *I took a look at the code for MatSetValues, if I am understanding it >>>> correctly(hopefully) I think I could do it, maybe modify it or create a new >>>> routine entirely for this pupose.* >>>> >>>> *i.e. MatSetValuesFromCSR(.....)* >>>> *Or is there a particular reason why it has to be this way* >>>> >>>> I also tried ksp ex3 but I slightly tweaked it to add a logging stage >>>> around the assembly and MatSetValues and I am attaching the modified >>>> example here as well. >>>> Although in this example the matrix stash is not empty ( means >>>> off-processor values are being set ) but the timing values for roughly the >>>> same matrix size , the command I used is >>>> mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info >>>> >>>> >>>> Regards, >>>> Kamra >>>> >>>> On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >>>>> mo7ammedmostafa at gmail.com> wrote: >>>>> >>>>>> Hello Matt, >>>>>> Attached is the dumped entire log output using -log_view and -info. >>>>>> >>>>> >>>>> In matrix construction, it looks like you have a mixture of load >>>>> imbalance (see the imbalance in the Begin events) >>>>> and lots of Scatter messages in your assembly. We turn off >>>>> MatSetValues() logging by default since it is usually >>>>> called many times, but you can explicitly turn it back on if you want. >>>>> I don't think that is the problem here. Its easy >>>>> to see from examples (say SNES ex5) that it is not the major time >>>>> sink. What is the Scatter doing? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Kamra >>>>>> >>>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>> >>>>>>>> Hello all, >>>>>>>> I have a few question regarding Petsc, >>>>>>>> >>>>>>> >>>>>>> Please send the entire output of a run with all the logging turned >>>>>>> on, using -log_view and -info. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Question 1: >>>>>>>> For the profiling , is it possible to only show the user defined >>>>>>>> log events in the breakdown of each stage in Log-view. >>>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>>>> every stage" according to the manual. >>>>>>>> however I still see them in the stage breakdown >>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>> >>>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>>> >>>>>>>> Also is possible to clear the logs so that I can write a separate >>>>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>>>> problem and I want to know the change in performance as time goes by ) >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>> Question 2: >>>>>>>> Regarding MatSetValues >>>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>>> in "Mat A" >>>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>>> but it is very slow and it is killing my performance >>>>>>>> although the matrix was properly set using >>>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>>>> PETSC_DETERMINE, >>>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>>>> So my question is it possible to set multiple rows at once >>>>>>>> hopefully all, I checked the manual and MatSetValues can only set dense >>>>>>>> matrix block because it seems that row by row is expensive >>>>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>>>> stash is 0 ) >>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>>> mallocs. >>>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>>>> space: 0 unneeded,743028 used >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,742972 used >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>> space: 0 unneeded,743093 used >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,743036 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,742938 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>> space: 0 unneeded,743049 used >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>>> space: 0 unneeded,685 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>>> space: 0 unneeded,649 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>>> space: 0 unneeded,1011 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>>> space: 0 unneeded,1137 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>>> space: 0 unneeded,658 used >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>>> space: 0 unneeded,648 used >>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>> >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>> Question 3: >>>>>>>> If all matrix and vector inserted data are local, what part of the >>>>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>>>> more time than matrix builder >>>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>>> >>>>>>>> >>>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned >>>>>>>> over six processes and it was NOT built using DM >>>>>>>> >>>>>>>> Finally the configure options are: >>>>>>>> >>>>>>>> Configure options: >>>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>>>> >>>>>>>> Sorry for such long question and thanks in advance >>>>>>>> Thanks >>>>>>>> M. Kamra >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 13 13:03:02 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jul 2019 14:03:02 -0400 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: You call the assembly stuff a lot (200). BuildTwoSidedF is a global thing and is taking a lot of time. You should just call these once per time step (it looks like you are just doing one time step). --- Event Stage 1: Matrix Construction BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello Matt, > Attached is the dumped entire log output using -log_view and -info. > > Thanks, > Kamra > > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley wrote: > >> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello all, >>> I have a few question regarding Petsc, >>> >> >> Please send the entire output of a run with all the logging turned on, >> using -log_view and -info. >> >> Thanks, >> >> Matt >> >> >>> Question 1: >>> For the profiling , is it possible to only show the user defined log >>> events in the breakdown of each stage in Log-view. >>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>> PetscLogEventExcludeClass(MAT_CLASSID); >>> PetscLogEventExcludeClass(VEC_CLASSID); >>> PetscLogEventExcludeClass(KSP_CLASSID); >>> PetscLogEventExcludeClass(PC_CLASSID); >>> which should "Deactivates event logging for a PETSc object class in >>> every stage" according to the manual. >>> however I still see them in the stage breakdown >>> --- Event Stage 1: Matrix Construction >>> >>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 >>> 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 >>> 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 >>> 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>> >>> Also is possible to clear the logs so that I can write a separate >>> profiling output file for each timestep ( since I am solving a transient >>> problem and I want to know the change in performance as time goes by ) >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> Question 2: >>> Regarding MatSetValues >>> Right now, I writing a finite volume code, due to algorithm requirement >>> I have to write the matrix into local native format ( array of arrays) and >>> then loop through rows and use MatSetValues to set the elements in "Mat A" >>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>> but it is very slow and it is killing my performance >>> although the matrix was properly set using >>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>> PETSC_DETERMINE, >>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>> matsetvalues and all inserted values are local so no off-processor values >>> So my question is it possible to set multiple rows at once hopefully >>> all, I checked the manual and MatSetValues can only set dense matrix block >>> because it seems that row by row is expensive >>> Or perhaps is it possible to copy all rows to the underlying matrix >>> data, as I mentioned all values are local and no off-processor values ( >>> stash is 0 ) >>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>> mallocs. >>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>> space: 0 unneeded,743028 used >>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742972 used >>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743093 used >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,743036 used >>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742938 used >>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743049 used >>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 >>> unneeded,685 used >>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 >>> unneeded,649 used >>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: >>> 0 unneeded,1011 used >>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: >>> 0 unneeded,1137 used >>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 >>> unneeded,658 used >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 >>> unneeded,648 used >>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> Question 3: >>> If all matrix and vector inserted data are local, what part of the >>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>> more time than matrix builder >>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>> >>> >>> For context the matrix in the above is nearly 1Mx1M partitioned over six >>> processes and it was NOT built using DM >>> >>> Finally the configure options are: >>> >>> Configure options: >>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>> --with-fc=mpif90 --download-metis --download-hypre >>> >>> Sorry for such long question and thanks in advance >>> Thanks >>> M. Kamra >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 13:07:42 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 03:07:42 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: This log is for 100 time-steps, not a single time step On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: > You call the assembly stuff a lot (200). BuildTwoSidedF is a global thing > and is taking a lot of time. You should just call these once per time step > (it looks like you are just doing one time step). > > > --- Event Stage 1: Matrix Construction > > BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 > VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 > VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 > VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 > AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 > myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 > setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 > setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 > callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 > > > > On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello Matt, >> Attached is the dumped entire log output using -log_view and -info. >> >> Thanks, >> Kamra >> >> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >> wrote: >> >>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello all, >>>> I have a few question regarding Petsc, >>>> >>> >>> Please send the entire output of a run with all the logging turned on, >>> using -log_view and -info. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Question 1: >>>> For the profiling , is it possible to only show the user defined log >>>> events in the breakdown of each stage in Log-view. >>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>> PetscLogEventExcludeClass(PC_CLASSID); >>>> which should "Deactivates event logging for a PETSc object class in >>>> every stage" according to the manual. >>>> however I still see them in the stage breakdown >>>> --- Event Stage 1: Matrix Construction >>>> >>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>> >>>> Also is possible to clear the logs so that I can write a separate >>>> profiling output file for each timestep ( since I am solving a transient >>>> problem and I want to know the change in performance as time goes by ) >>>> >>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> Question 2: >>>> Regarding MatSetValues >>>> Right now, I writing a finite volume code, due to algorithm requirement >>>> I have to write the matrix into local native format ( array of arrays) and >>>> then loop through rows and use MatSetValues to set the elements in "Mat A" >>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>> but it is very slow and it is killing my performance >>>> although the matrix was properly set using >>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>> PETSC_DETERMINE, >>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>> matsetvalues and all inserted values are local so no off-processor values >>>> So my question is it possible to set multiple rows at once hopefully >>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>> because it seems that row by row is expensive >>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>> data, as I mentioned all values are local and no off-processor values ( >>>> stash is 0 ) >>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>> mallocs. >>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>> space: 0 unneeded,743028 used >>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,742972 used >>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>> space: 0 unneeded,743093 used >>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,743036 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>> space: 0 unneeded,742938 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>> space: 0 unneeded,743049 used >>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >>>> 0 unneeded,685 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >>>> 0 unneeded,649 used >>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: >>>> 0 unneeded,1011 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: >>>> 0 unneeded,1137 used >>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >>>> 0 unneeded,658 used >>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >>>> 0 unneeded,648 used >>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >>>> 0 >>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>> >>>> >>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> Question 3: >>>> If all matrix and vector inserted data are local, what part of the >>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>> more time than matrix builder >>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>> >>>> >>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>> six processes and it was NOT built using DM >>>> >>>> Finally the configure options are: >>>> >>>> Configure options: >>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>> --with-fc=mpif90 --download-metis --download-hypre >>>> >>>> Sorry for such long question and thanks in advance >>>> Thanks >>>> M. Kamra >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 13 13:35:27 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jul 2019 14:35:27 -0400 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Ok, I only see one all to KSPSolve. On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa wrote: > This log is for 100 time-steps, not a single time step > > > On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: > >> You call the assembly stuff a lot (200). BuildTwoSidedF is a global thing >> and is taking a lot of time. You should just call these once per time step >> (it looks like you are just doing one time step). >> >> >> --- Event Stage 1: Matrix Construction >> >> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >> >> >> >> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello Matt, >>> Attached is the dumped entire log output using -log_view and -info. >>> >>> Thanks, >>> Kamra >>> >>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>> wrote: >>> >>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Hello all, >>>>> I have a few question regarding Petsc, >>>>> >>>> >>>> Please send the entire output of a run with all the logging turned on, >>>> using -log_view and -info. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Question 1: >>>>> For the profiling , is it possible to only show the user defined log >>>>> events in the breakdown of each stage in Log-view. >>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>> which should "Deactivates event logging for a PETSc object class in >>>>> every stage" according to the manual. >>>>> however I still see them in the stage breakdown >>>>> --- Event Stage 1: Matrix Construction >>>>> >>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>> >>>>> Also is possible to clear the logs so that I can write a separate >>>>> profiling output file for each timestep ( since I am solving a transient >>>>> problem and I want to know the change in performance as time goes by ) >>>>> >>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> Question 2: >>>>> Regarding MatSetValues >>>>> Right now, I writing a finite volume code, due to algorithm >>>>> requirement I have to write the matrix into local native format ( array of >>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>> in "Mat A" >>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>> but it is very slow and it is killing my performance >>>>> although the matrix was properly set using >>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>> PETSC_DETERMINE, >>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>> matsetvalues and all inserted values are local so no off-processor values >>>>> So my question is it possible to set multiple rows at once hopefully >>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>> because it seems that row by row is expensive >>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>> data, as I mentioned all values are local and no off-processor values ( >>>>> stash is 0 ) >>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>> mallocs. >>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>> space: 0 unneeded,743028 used >>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,742972 used >>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>> space: 0 unneeded,743093 used >>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,743036 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>> space: 0 unneeded,742938 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>> space: 0 unneeded,743049 used >>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >>>>> 0 unneeded,685 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >>>>> 0 unneeded,649 used >>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>> space: 0 unneeded,1011 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>> space: 0 unneeded,1137 used >>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >>>>> 0 unneeded,658 used >>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >>>>> 0 unneeded,648 used >>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>> is 0 >>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>> >>>>> >>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> Question 3: >>>>> If all matrix and vector inserted data are local, what part of the >>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>> more time than matrix builder >>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>> >>>>> >>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>> six processes and it was NOT built using DM >>>>> >>>>> Finally the configure options are: >>>>> >>>>> Configure options: >>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>> >>>>> Sorry for such long question and thanks in advance >>>>> Thanks >>>>> M. Kamra >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 13:37:20 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 03:37:20 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: <3BFD3707-6543-467A-8A1C-17A27B1A306C@anl.gov> References: <3BFD3707-6543-467A-8A1C-17A27B1A306C@anl.gov> Message-ID: I am generating the matrix using the finite volume method I basically loop over the face list instead of looping over the cells to avoid double evaluation of the fluxes of cell faces So I figured I would store the coefficients in a temp container ( in this case a csr sparse matrix) and then loop over rows to set in the petsc matrix I know it looks like a waste of memory and copy overhead but for now I can?t think of a better way. For now I will try the two routines from the master branch MatCreateMPIAIJWithArrays() MatUpdateMPIAIJWithArrays() and send the logs Thanks Kamra On Sun, Jul 14, 2019 at 1:51 AM Smith, Barry F. wrote: > > How are you generating entries in your matrix? Finite differences, > finite element, finite volume, something else? > > If you are using finite differences you generally generate an entire row > at a time and call MatSetValues() once per row. With finite elements you > generates an element at a time and ADD_VALUES for a block of rows and > columns. > > I don't know why generating directly in CSR format would be faster than > than calling MatSetValues() once per row but anyways if you have the matrix > in CSR format you can use > > MatCreateMPIAIJWithArrays() (and in the master branch of the repository) > MatUpdateMPIAIJWithArrays(). > > to build the matrix the first time, and then "refill" it with numerical > values each new time. There are a few other optimizations related to matrix > insertion in the master branch you might also benefit from. > > Generally for problems with multiple "times" or "linear solve steps" we > use two stages, the first to track the initial set up and first time step > and the other to capture all the other steps (since the extra overhead is > only in the first step.) You could make a new stage for each time step but > I don't think that is needed. > > After you have this going send us the new log summary. > > Barry > > > > > On Jul 13, 2019, at 11:20 AM, Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > I am sorry but I don?t see what you mean by small times > > Although mat assembly is relatively smaller > > The cost of mat set values is still significant > > The same can be said for vec assembly > > Combined vec/mat assembly and matsetvalues constitute about 50% of the > total cost of matrix construction > > > > So is this problem of my matrix setup/ preallocation > > > > Or is this a hardware issue, for whatever reason the copy is overly slow > > The code was run on a single node > > > > Or is this function call overhead since matsetvalues is being called 1M > times inside the for loop ( 170k times in each process) > > > > Thanks, Kamra > > > > On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley > wrote: > > On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > > Hello Matt, > > > > I revised my code and changed the way I create the rhs vector, > > previosly I was using vecCreateGhost just in case I need the ghost > values, but for now I changed that to > > vecCreateMPI(.......) > > So maybe that was the cause of the scatter > > I am attaching with this email a new log output > > > > Okay, the times are now very small. How does it scale up? > > > > Thanks, > > > > Matt > > > > Also regarding how I fill my petsc matrix, > > In my code I fill a temp CSR format matrix becasue otherwise I would > need "MatSetValue" to fill the petsc mat element by element > > which is not recommmeded in the petsc manual and probably very expensive > due to function call overhead > > So after I create my matrix in CSR format, I fill the PETSC mat A as > follows > > for (i = 0; i < nMatRows; i++) { > > cffset = CSR_iptr[i]; > > row_index = row_gIndex[i]; > > nj = Eqn_nj[i]; > > MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, > CSR_vptr + offset, INSERT_VALUES); > > } > > After That > > VecAssemblyBegin(RHS); > > VecAssemblyEnd(RHS); > > > > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > > I don't believe , I am doing anything special, if possible I would like > to set the whole csr matrix at once in one command. > > I took a look at the code for MatSetValues, if I am understanding it > correctly(hopefully) I think I could do it, maybe modify it or create a new > routine entirely for this pupose. > > i.e. MatSetValuesFromCSR(.....) > > Or is there a particular reason why it has to be this way > > > > I also tried ksp ex3 but I slightly tweaked it to add a logging stage > around the assembly and MatSetValues and I am attaching the modified > example here as well. > > Although in this example the matrix stash is not empty ( means > off-processor values are being set ) but the timing values for roughly the > same matrix size , the command I used is > > mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info > > > > > > Regards, > > Kamra > > > > On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley > wrote: > > On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > > Hello Matt, > > Attached is the dumped entire log output using -log_view and -info. > > > > In matrix construction, it looks like you have a mixture of load > imbalance (see the imbalance in the Begin events) > > and lots of Scatter messages in your assembly. We turn off > MatSetValues() logging by default since it is usually > > called many times, but you can explicitly turn it back on if you want. I > don't think that is the problem here. Its easy > > to see from examples (say SNES ex5) that it is not the major time sink. > What is the Scatter doing? > > > > Thanks, > > > > Matt > > > > Thanks, > > Kamra > > > > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley > wrote: > > On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hello all, > > I have a few question regarding Petsc, > > > > Please send the entire output of a run with all the logging turned on, > using -log_view and -info. > > > > Thanks, > > > > Matt > > > > Question 1: > > For the profiling , is it possible to only show the user defined log > events in the breakdown of each stage in Log-view. > > I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, > > PetscLogEventExcludeClass(MAT_CLASSID); > > PetscLogEventExcludeClass(VEC_CLASSID); > > PetscLogEventExcludeClass(KSP_CLASSID); > > PetscLogEventExcludeClass(PC_CLASSID); > > which should "Deactivates event logging for a PETSc object class in > every stage" according to the manual. > > however I still see them in the stage breakdown > > --- Event Stage 1: Matrix Construction > > > > BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > > VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 > > VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 > 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 > > VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 > 8.0e+00 0 0 3 0 6 10 0 50 20100 0 > > AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 > 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT > > myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT > > setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT > > setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT > > callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT > > > > Also is possible to clear the logs so that I can write a separate > profiling output file for each timestep ( since I am solving a transient > problem and I want to know the change in performance as time goes by ) > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Question 2: > > Regarding MatSetValues > > Right now, I writing a finite volume code, due to algorithm requirement > I have to write the matrix into local native format ( array of arrays) and > then loop through rows and use MatSetValues to set the elements in "Mat A" > > MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); > > but it is very slow and it is killing my performance > > although the matrix was properly set using > > MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, > PETSC_DETERMINE, > > PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); > > with d_nnz,and o_nnz properly assigned so no mallocs occur during > matsetvalues and all inserted values are local so no off-processor values > > So my question is it possible to set multiple rows at once hopefully > all, I checked the manual and MatSetValues can only set dense matrix block > because it seems that row by row is expensive > > Or perhaps is it possible to copy all rows to the underlying matrix > data, as I mentioned all values are local and no off-processor values ( > stash is 0 ) > > [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. > > [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 > mallocs. > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage > space: 0 unneeded,743028 used > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage > space: 0 unneeded,742972 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage > space: 0 unneeded,743093 used > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage > space: 0 unneeded,743036 used > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage > space: 0 unneeded,742938 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. > > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage > space: 0 unneeded,743049 used > > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 > > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. > > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 > unneeded,685 used > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 > unneeded,649 used > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: > 0 unneeded,1011 used > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: > 0 unneeded,1137 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 > unneeded,658 used > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 > unneeded,648 used > > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. > > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 > > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows > 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Question 3: > > If all matrix and vector inserted data are local, what part of the > vec/mat assembly consumes time because matsetvalues and matassembly consume > more time than matrix builder > > Also this is not just for the first time MAT_FINAL_ASSEMBLY > > > > > > For context the matrix in the above is nearly 1Mx1M partitioned over six > processes and it was NOT built using DM > > > > Finally the configure options are: > > > > Configure options: > > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native > -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 > -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx > --with-fc=mpif90 --download-metis --download-hypre > > > > Sorry for such long question and thanks in advance > > Thanks > > M. Kamra > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 13:43:08 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 03:43:08 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Sorry about that I wanted to see if the assembly cost would drop with subsequent time steps but it was taking too long to run so I set it to solve only once since I was only interested in profiling the matrix builder. Again sorry for that Kamra On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: > Ok, I only see one all to KSPSolve. > > On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> This log is for 100 time-steps, not a single time step >> >> >> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >> >>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>> thing and is taking a lot of time. You should just call these once per time >>> step (it looks like you are just doing one time step). >>> >>> >>> --- Event Stage 1: Matrix Construction >>> >>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>> >>> >>> >>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello Matt, >>>> Attached is the dumped entire log output using -log_view and -info. >>>> >>>> Thanks, >>>> Kamra >>>> >>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>> petsc-users at mcs.anl.gov> wrote: >>>>> >>>>>> Hello all, >>>>>> I have a few question regarding Petsc, >>>>>> >>>>> >>>>> Please send the entire output of a run with all the logging turned on, >>>>> using -log_view and -info. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Question 1: >>>>>> For the profiling , is it possible to only show the user defined log >>>>>> events in the breakdown of each stage in Log-view. >>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>> every stage" according to the manual. >>>>>> however I still see them in the stage breakdown >>>>>> --- Event Stage 1: Matrix Construction >>>>>> >>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>> >>>>>> Also is possible to clear the logs so that I can write a separate >>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>> problem and I want to know the change in performance as time goes by ) >>>>>> >>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>> Question 2: >>>>>> Regarding MatSetValues >>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>> requirement I have to write the matrix into local native format ( array of >>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>> in "Mat A" >>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>> but it is very slow and it is killing my performance >>>>>> although the matrix was properly set using >>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>> PETSC_DETERMINE, >>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>> So my question is it possible to set multiple rows at once hopefully >>>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>>> because it seems that row by row is expensive >>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>> stash is 0 ) >>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>> mallocs. >>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>> space: 0 unneeded,743028 used >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,742972 used >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>> space: 0 unneeded,743093 used >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,743036 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>> space: 0 unneeded,742938 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>> space: 0 unneeded,743049 used >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>> space: 0 unneeded,685 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>> space: 0 unneeded,649 used >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>> space: 0 unneeded,1011 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>> space: 0 unneeded,1137 used >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>> space: 0 unneeded,658 used >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>> space: 0 unneeded,648 used >>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>> is 0 >>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>> >>>>>> >>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>> Question 3: >>>>>> If all matrix and vector inserted data are local, what part of the >>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>> more time than matrix builder >>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>> >>>>>> >>>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>>> six processes and it was NOT built using DM >>>>>> >>>>>> Finally the configure options are: >>>>>> >>>>>> Configure options: >>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>> >>>>>> Sorry for such long question and thanks in advance >>>>>> Thanks >>>>>> M. Kamra >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 13 13:57:20 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jul 2019 14:57:20 -0400 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: <3BFD3707-6543-467A-8A1C-17A27B1A306C@anl.gov> Message-ID: On Sat, Jul 13, 2019 at 2:39 PM Mohammed Mostafa via petsc-users < petsc-users at mcs.anl.gov> wrote: > I am generating the matrix using the finite volume method > I basically loop over the face list instead of looping over the cells to > avoid double evaluation of the fluxes of cell faces > So I figured I would store the coefficients in a temp container ( in this > case a csr sparse matrix) and then loop over rows to set in the petsc > matrix > I know it looks like a waste of memory and copy overhead but for now I > can?t think of a better way. > You can use PETSc's setValues, just use ADD_VALUES instead of INSERT. Our matrix class is pretty much the same as your "temp container" so I would use PETSc directly. And I see "Maximum nonzeros in any row is 4" and "Maximum nonzeros in any row is 1". I guess the "1"s are from the off diagonal block matrix, but should the 4 be 2*D + 1 ? > For now I will try the two routines from the master branch > MatCreateMPIAIJWithArrays() MatUpdateMPIAIJWithArrays() > and send the logs > > Thanks > Kamra > > On Sun, Jul 14, 2019 at 1:51 AM Smith, Barry F. > wrote: > >> >> How are you generating entries in your matrix? Finite differences, >> finite element, finite volume, something else? >> >> If you are using finite differences you generally generate an entire >> row at a time and call MatSetValues() once per row. With finite elements >> you generates an element at a time and ADD_VALUES for a block of rows and >> columns. >> >> I don't know why generating directly in CSR format would be faster than >> than calling MatSetValues() once per row but anyways if you have the matrix >> in CSR format you can use >> >> MatCreateMPIAIJWithArrays() (and in the master branch of the >> repository) MatUpdateMPIAIJWithArrays(). >> >> to build the matrix the first time, and then "refill" it with numerical >> values each new time. There are a few other optimizations related to matrix >> insertion in the master branch you might also benefit from. >> >> Generally for problems with multiple "times" or "linear solve steps" we >> use two stages, the first to track the initial set up and first time step >> and the other to capture all the other steps (since the extra overhead is >> only in the first step.) You could make a new stage for each time step but >> I don't think that is needed. >> >> After you have this going send us the new log summary. >> >> Barry >> >> >> >> > On Jul 13, 2019, at 11:20 AM, Mohammed Mostafa via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > >> > I am sorry but I don?t see what you mean by small times >> > Although mat assembly is relatively smaller >> > The cost of mat set values is still significant >> > The same can be said for vec assembly >> > Combined vec/mat assembly and matsetvalues constitute about 50% of the >> total cost of matrix construction >> > >> > So is this problem of my matrix setup/ preallocation >> > >> > Or is this a hardware issue, for whatever reason the copy is overly >> slow >> > The code was run on a single node >> > >> > Or is this function call overhead since matsetvalues is being called 1M >> times inside the for loop ( 170k times in each process) >> > >> > Thanks, Kamra >> > >> > On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley >> wrote: >> > On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> > Hello Matt, >> > >> > I revised my code and changed the way I create the rhs vector, >> > previosly I was using vecCreateGhost just in case I need the ghost >> values, but for now I changed that to >> > vecCreateMPI(.......) >> > So maybe that was the cause of the scatter >> > I am attaching with this email a new log output >> > >> > Okay, the times are now very small. How does it scale up? >> > >> > Thanks, >> > >> > Matt >> > >> > Also regarding how I fill my petsc matrix, >> > In my code I fill a temp CSR format matrix becasue otherwise I would >> need "MatSetValue" to fill the petsc mat element by element >> > which is not recommmeded in the petsc manual and probably very >> expensive due to function call overhead >> > So after I create my matrix in CSR format, I fill the PETSC mat A as >> follows >> > for (i = 0; i < nMatRows; i++) { >> > cffset = CSR_iptr[i]; >> > row_index = row_gIndex[i]; >> > nj = Eqn_nj[i]; >> > MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >> CSR_vptr + offset, INSERT_VALUES); >> > } >> > After That >> > VecAssemblyBegin(RHS); >> > VecAssemblyEnd(RHS); >> > >> > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> > >> > I don't believe , I am doing anything special, if possible I would like >> to set the whole csr matrix at once in one command. >> > I took a look at the code for MatSetValues, if I am understanding it >> correctly(hopefully) I think I could do it, maybe modify it or create a new >> routine entirely for this pupose. >> > i.e. MatSetValuesFromCSR(.....) >> > Or is there a particular reason why it has to be this way >> > >> > I also tried ksp ex3 but I slightly tweaked it to add a logging stage >> around the assembly and MatSetValues and I am attaching the modified >> example here as well. >> > Although in this example the matrix stash is not empty ( means >> off-processor values are being set ) but the timing values for roughly the >> same matrix size , the command I used is >> > mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info >> > >> > >> > Regards, >> > Kamra >> > >> > On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley >> wrote: >> > On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> > Hello Matt, >> > Attached is the dumped entire log output using -log_view and -info. >> > >> > In matrix construction, it looks like you have a mixture of load >> imbalance (see the imbalance in the Begin events) >> > and lots of Scatter messages in your assembly. We turn off >> MatSetValues() logging by default since it is usually >> > called many times, but you can explicitly turn it back on if you want. >> I don't think that is the problem here. Its easy >> > to see from examples (say SNES ex5) that it is not the major time sink. >> What is the Scatter doing? >> > >> > Thanks, >> > >> > Matt >> > >> > Thanks, >> > Kamra >> > >> > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >> wrote: >> > On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > Hello all, >> > I have a few question regarding Petsc, >> > >> > Please send the entire output of a run with all the logging turned on, >> using -log_view and -info. >> > >> > Thanks, >> > >> > Matt >> > >> > Question 1: >> > For the profiling , is it possible to only show the user defined log >> events in the breakdown of each stage in Log-view. >> > I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >> > PetscLogEventExcludeClass(MAT_CLASSID); >> > PetscLogEventExcludeClass(VEC_CLASSID); >> > PetscLogEventExcludeClass(KSP_CLASSID); >> > PetscLogEventExcludeClass(PC_CLASSID); >> > which should "Deactivates event logging for a PETSc object class in >> every stage" according to the manual. >> > however I still see them in the stage breakdown >> > --- Event Stage 1: Matrix Construction >> > >> > BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >> > VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >> > VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >> > VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >> > AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >> > myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >> > setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >> > setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >> > callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >> > >> > Also is possible to clear the logs so that I can write a separate >> profiling output file for each timestep ( since I am solving a transient >> problem and I want to know the change in performance as time goes by ) >> > >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> > Question 2: >> > Regarding MatSetValues >> > Right now, I writing a finite volume code, due to algorithm requirement >> I have to write the matrix into local native format ( array of arrays) and >> then loop through rows and use MatSetValues to set the elements in "Mat A" >> > MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >> > but it is very slow and it is killing my performance >> > although the matrix was properly set using >> > MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >> PETSC_DETERMINE, >> > PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >> > with d_nnz,and o_nnz properly assigned so no mallocs occur during >> matsetvalues and all inserted values are local so no off-processor values >> > So my question is it possible to set multiple rows at once hopefully >> all, I checked the manual and MatSetValues can only set dense matrix block >> because it seems that row by row is expensive >> > Or perhaps is it possible to copy all rows to the underlying matrix >> data, as I mentioned all values are local and no off-processor values ( >> stash is 0 ) >> > [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >> > [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >> mallocs. >> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >> > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >> space: 0 unneeded,743028 used >> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >> space: 0 unneeded,742972 used >> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >> space: 0 unneeded,743093 used >> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >> space: 0 unneeded,743036 used >> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >> space: 0 unneeded,742938 used >> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >> > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >> space: 0 unneeded,743049 used >> > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >> > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >> > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >> 0 unneeded,685 used >> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >> 0 unneeded,649 used >> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: >> 0 unneeded,1011 used >> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: >> 0 unneeded,1137 used >> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >> 0 unneeded,658 used >> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >> 0 unneeded,648 used >> > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >> > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is >> 0 >> > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >> > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >> > >> > >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> > Question 3: >> > If all matrix and vector inserted data are local, what part of the >> vec/mat assembly consumes time because matsetvalues and matassembly consume >> more time than matrix builder >> > Also this is not just for the first time MAT_FINAL_ASSEMBLY >> > >> > >> > For context the matrix in the above is nearly 1Mx1M partitioned over >> six processes and it was NOT built using DM >> > >> > Finally the configure options are: >> > >> > Configure options: >> > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >> --with-fc=mpif90 --download-metis --download-hypre >> > >> > Sorry for such long question and thanks in advance >> > Thanks >> > M. Kamra >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 13 14:02:19 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 14 Jul 2019 04:02:19 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: <3BFD3707-6543-467A-8A1C-17A27B1A306C@anl.gov> Message-ID: This is a 2D case on triangular mesh so 3+1 For quad it becomes 4+1 and in 3D more and more nnz depending on the cell type On Sun, Jul 14, 2019 at 3:55 AM Mark Adams wrote: > > > On Sat, Jul 13, 2019 at 2:39 PM Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> I am generating the matrix using the finite volume method >> I basically loop over the face list instead of looping over the cells to >> avoid double evaluation of the fluxes of cell faces >> So I figured I would store the coefficients in a temp container ( in this >> case a csr sparse matrix) and then loop over rows to set in the petsc >> matrix >> I know it looks like a waste of memory and copy overhead but for now I >> can?t think of a better way. >> > > You can use PETSc's setValues, just use ADD_VALUES instead of INSERT. Our > matrix class is pretty much the same as your "temp container" so I would > use PETSc directly. > > And I see "Maximum nonzeros in any row is 4" and "Maximum nonzeros in any > row is 1". I guess the "1"s are from the off diagonal block matrix, but > should the 4 be 2*D + 1 ? > > >> For now I will try the two routines from the master branch >> MatCreateMPIAIJWithArrays() MatUpdateMPIAIJWithArrays() >> and send the logs >> >> Thanks >> Kamra >> >> On Sun, Jul 14, 2019 at 1:51 AM Smith, Barry F. >> wrote: >> >>> >>> How are you generating entries in your matrix? Finite differences, >>> finite element, finite volume, something else? >>> >>> If you are using finite differences you generally generate an entire >>> row at a time and call MatSetValues() once per row. With finite elements >>> you generates an element at a time and ADD_VALUES for a block of rows and >>> columns. >>> >>> I don't know why generating directly in CSR format would be faster >>> than than calling MatSetValues() once per row but anyways if you have the >>> matrix in CSR format you can use >>> >>> MatCreateMPIAIJWithArrays() (and in the master branch of the >>> repository) MatUpdateMPIAIJWithArrays(). >>> >>> to build the matrix the first time, and then "refill" it with numerical >>> values each new time. There are a few other optimizations related to matrix >>> insertion in the master branch you might also benefit from. >>> >>> Generally for problems with multiple "times" or "linear solve steps" >>> we use two stages, the first to track the initial set up and first time >>> step and the other to capture all the other steps (since the extra overhead >>> is only in the first step.) You could make a new stage for each time step >>> but I don't think that is needed. >>> >>> After you have this going send us the new log summary. >>> >>> Barry >>> >>> >>> >>> > On Jul 13, 2019, at 11:20 AM, Mohammed Mostafa via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> > >>> > I am sorry but I don?t see what you mean by small times >>> > Although mat assembly is relatively smaller >>> > The cost of mat set values is still significant >>> > The same can be said for vec assembly >>> > Combined vec/mat assembly and matsetvalues constitute about 50% of >>> the total cost of matrix construction >>> > >>> > So is this problem of my matrix setup/ preallocation >>> > >>> > Or is this a hardware issue, for whatever reason the copy is overly >>> slow >>> > The code was run on a single node >>> > >>> > Or is this function call overhead since matsetvalues is being called >>> 1M times inside the for loop ( 170k times in each process) >>> > >>> > Thanks, Kamra >>> > >>> > On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley >>> wrote: >>> > On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> > Hello Matt, >>> > >>> > I revised my code and changed the way I create the rhs vector, >>> > previosly I was using vecCreateGhost just in case I need the ghost >>> values, but for now I changed that to >>> > vecCreateMPI(.......) >>> > So maybe that was the cause of the scatter >>> > I am attaching with this email a new log output >>> > >>> > Okay, the times are now very small. How does it scale up? >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > Also regarding how I fill my petsc matrix, >>> > In my code I fill a temp CSR format matrix becasue otherwise I would >>> need "MatSetValue" to fill the petsc mat element by element >>> > which is not recommmeded in the petsc manual and probably very >>> expensive due to function call overhead >>> > So after I create my matrix in CSR format, I fill the PETSC mat A as >>> follows >>> > for (i = 0; i < nMatRows; i++) { >>> > cffset = CSR_iptr[i]; >>> > row_index = row_gIndex[i]; >>> > nj = Eqn_nj[i]; >>> > MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, >>> CSR_vptr + offset, INSERT_VALUES); >>> > } >>> > After That >>> > VecAssemblyBegin(RHS); >>> > VecAssemblyEnd(RHS); >>> > >>> > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> > >>> > I don't believe , I am doing anything special, if possible I would >>> like to set the whole csr matrix at once in one command. >>> > I took a look at the code for MatSetValues, if I am understanding it >>> correctly(hopefully) I think I could do it, maybe modify it or create a new >>> routine entirely for this pupose. >>> > i.e. MatSetValuesFromCSR(.....) >>> > Or is there a particular reason why it has to be this way >>> > >>> > I also tried ksp ex3 but I slightly tweaked it to add a logging stage >>> around the assembly and MatSetValues and I am attaching the modified >>> example here as well. >>> > Although in this example the matrix stash is not empty ( means >>> off-processor values are being set ) but the timing values for roughly the >>> same matrix size , the command I used is >>> > mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info >>> > >>> > >>> > Regards, >>> > Kamra >>> > >>> > On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley >>> wrote: >>> > On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> > Hello Matt, >>> > Attached is the dumped entire log output using -log_view and -info. >>> > >>> > In matrix construction, it looks like you have a mixture of load >>> imbalance (see the imbalance in the Begin events) >>> > and lots of Scatter messages in your assembly. We turn off >>> MatSetValues() logging by default since it is usually >>> > called many times, but you can explicitly turn it back on if you want. >>> I don't think that is the problem here. Its easy >>> > to see from examples (say SNES ex5) that it is not the major time >>> sink. What is the Scatter doing? >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > Thanks, >>> > Kamra >>> > >>> > On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>> wrote: >>> > On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> > Hello all, >>> > I have a few question regarding Petsc, >>> > >>> > Please send the entire output of a run with all the logging turned on, >>> using -log_view and -info. >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > Question 1: >>> > For the profiling , is it possible to only show the user defined log >>> events in the breakdown of each stage in Log-view. >>> > I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>> > PetscLogEventExcludeClass(MAT_CLASSID); >>> > PetscLogEventExcludeClass(VEC_CLASSID); >>> > PetscLogEventExcludeClass(KSP_CLASSID); >>> > PetscLogEventExcludeClass(PC_CLASSID); >>> > which should "Deactivates event logging for a PETSc object class in >>> every stage" according to the manual. >>> > however I still see them in the stage breakdown >>> > --- Event Stage 1: Matrix Construction >>> > >>> > BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> > VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> > VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>> > VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> > VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>> > VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> > MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> > MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>> > AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>> > myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>> > setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>> > setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>> > callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>> > >>> > Also is possible to clear the logs so that I can write a separate >>> profiling output file for each timestep ( since I am solving a transient >>> problem and I want to know the change in performance as time goes by ) >>> > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> > Question 2: >>> > Regarding MatSetValues >>> > Right now, I writing a finite volume code, due to algorithm >>> requirement I have to write the matrix into local native format ( array of >>> arrays) and then loop through rows and use MatSetValues to set the elements >>> in "Mat A" >>> > MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>> > but it is very slow and it is killing my performance >>> > although the matrix was properly set using >>> > MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>> PETSC_DETERMINE, >>> > PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>> > with d_nnz,and o_nnz properly assigned so no mallocs occur during >>> matsetvalues and all inserted values are local so no off-processor values >>> > So my question is it possible to set multiple rows at once hopefully >>> all, I checked the manual and MatSetValues can only set dense matrix block >>> because it seems that row by row is expensive >>> > Or perhaps is it possible to copy all rows to the underlying matrix >>> data, as I mentioned all values are local and no off-processor values ( >>> stash is 0 ) >>> > [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>> > [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>> mallocs. >>> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>> space: 0 unneeded,743028 used >>> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742972 used >>> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743093 used >>> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,743036 used >>> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>> space: 0 unneeded,742938 used >>> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>> > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>> space: 0 unneeded,743049 used >>> > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>> > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>> > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: >>> 0 unneeded,685 used >>> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: >>> 0 unneeded,649 used >>> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>> space: 0 unneeded,1011 used >>> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>> space: 0 unneeded,1137 used >>> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: >>> 0 unneeded,658 used >>> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: >>> 0 unneeded,648 used >>> > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>> > [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>> > [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>> > >>> > >>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> > Question 3: >>> > If all matrix and vector inserted data are local, what part of the >>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>> more time than matrix builder >>> > Also this is not just for the first time MAT_FINAL_ASSEMBLY >>> > >>> > >>> > For context the matrix in the above is nearly 1Mx1M partitioned over >>> six processes and it was NOT built using DM >>> > >>> > Finally the configure options are: >>> > >>> > Configure options: >>> > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>> --with-fc=mpif90 --download-metis --download-hypre >>> > >>> > Sorry for such long question and thanks in advance >>> > Thanks >>> > M. Kamra >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://www.cse.buffalo.edu/~knepley/ >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://www.cse.buffalo.edu/~knepley/ >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://www.cse.buffalo.edu/~knepley/ >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 13 14:19:14 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 13 Jul 2019 15:19:14 -0400 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for its own rows, will generate an error if any process sets values for another process. This avoids all reductions in the MatAssembly routines and thus improves performance for very large process counts. OK, so I am seeing the whole matrix creation, including your flux calcs and your intermediate data structure, as taking 0.1 sec (10/100). That is about 7% of the solve time (but this looks like it could use some attention) or about 25 Mat-vecs (the standard work unit of an iterative solver). Now I see 10% of the matrix creation time going to MatAssembly end, which is annoying because there is no communication. I don't see any big problems here, except for the solver maybe, if this is a nice friendly Laplacian. * I would try PETSc's Set Value routines directly. * You might as well try https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetOption.html with MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for its own rows, will generate an error if any process sets values for another process. This avoids all reductions in the MatAssembly routines and thus improves performance for very large process counts. This should eliminate the MatAssemblyEnd cost. On Sat, Jul 13, 2019 at 2:43 PM Mohammed Mostafa wrote: > Sorry about that > I wanted to see if the assembly cost would drop with subsequent time steps > but it was taking too long to run so I set it to solve only once since I > was only interested in profiling the matrix builder. > Again sorry for that > Kamra > > On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: > >> Ok, I only see one all to KSPSolve. >> >> On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> This log is for 100 time-steps, not a single time step >>> >>> >>> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >>> >>>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>>> thing and is taking a lot of time. You should just call these once per time >>>> step (it looks like you are just doing one time step). >>>> >>>> >>>> --- Event Stage 1: Matrix Construction >>>> >>>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>>> >>>> >>>> >>>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Hello Matt, >>>>> Attached is the dumped entire log output using -log_view and -info. >>>>> >>>>> Thanks, >>>>> Kamra >>>>> >>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>> >>>>>>> Hello all, >>>>>>> I have a few question regarding Petsc, >>>>>>> >>>>>> >>>>>> Please send the entire output of a run with all the logging turned >>>>>> on, using -log_view and -info. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Question 1: >>>>>>> For the profiling , is it possible to only show the user defined log >>>>>>> events in the breakdown of each stage in Log-view. >>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>>> every stage" according to the manual. >>>>>>> however I still see them in the stage breakdown >>>>>>> --- Event Stage 1: Matrix Construction >>>>>>> >>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>> >>>>>>> Also is possible to clear the logs so that I can write a separate >>>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>>> problem and I want to know the change in performance as time goes by ) >>>>>>> >>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>> Question 2: >>>>>>> Regarding MatSetValues >>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>> in "Mat A" >>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>> but it is very slow and it is killing my performance >>>>>>> although the matrix was properly set using >>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>>> PETSC_DETERMINE, >>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>>> So my question is it possible to set multiple rows at once hopefully >>>>>>> all, I checked the manual and MatSetValues can only set dense matrix block >>>>>>> because it seems that row by row is expensive >>>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>>> stash is 0 ) >>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>> mallocs. >>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>>> space: 0 unneeded,743028 used >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,742972 used >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>> space: 0 unneeded,743093 used >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,743036 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>> space: 0 unneeded,742938 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>> space: 0 unneeded,743049 used >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>> space: 0 unneeded,685 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>> space: 0 unneeded,649 used >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>> space: 0 unneeded,1011 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>> space: 0 unneeded,1137 used >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>> space: 0 unneeded,658 used >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>> space: 0 unneeded,648 used >>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>>>>> is 0 >>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>> Question 3: >>>>>>> If all matrix and vector inserted data are local, what part of the >>>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>>> more time than matrix builder >>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>> >>>>>>> >>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned over >>>>>>> six processes and it was NOT built using DM >>>>>>> >>>>>>> Finally the configure options are: >>>>>>> >>>>>>> Configure options: >>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>>> >>>>>>> Sorry for such long question and thanks in advance >>>>>>> Thanks >>>>>>> M. Kamra >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From yjwu16 at gmail.com Sun Jul 14 05:22:03 2019 From: yjwu16 at gmail.com (Yingjie Wu) Date: Sun, 14 Jul 2019 18:22:03 +0800 Subject: [petsc-users] Questions about AMG and Jacobian Contruction Message-ID: Respected PETSc developers: Hi, I have some questions about some functions of AMG and the construction time of Jacobian matrix in the process of using. Please help me to answer them. 1. I see some functions about AMG in the list of PETSc functions. I wonder if DMDA is necessary to construct vectors when using AMG precondition. (Because the geometry described in my PDE is irregular,it contains a lot of pipes and connections, instead of using DMDA to manage vectors, I define discrete solution vectors by myself.) 2. As far as I know, besides V-cycle, AMG method also needs auxiliary matrix B and transfer matrix P (transfer matrix from fine mesh to coarse mesh). Are these transformation matrices built-in or can they be provided by users in PETSc? 3. At present, in my program, I use -snes_mf_operator to solve the non-linear PDE problem. Although I do not directly construct Jacobian matrix, it is also expensive to construct preconditioning matrix (provided by users). Can I control the update and decomposition time of the preconditioning matrix, such as updating the preconditioning matrix every two Newton steps and then ILU factorization? Can these controls be implemented in SNESMonitor? Thanks, Yingjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Jul 14 09:12:27 2019 From: jed at jedbrown.org (Jed Brown) Date: Sun, 14 Jul 2019 08:12:27 -0600 Subject: [petsc-users] Questions about AMG and Jacobian Contruction In-Reply-To: References: Message-ID: <87pnmciu2c.fsf@jedbrown.org> Yingjie Wu via petsc-users writes: > Respected PETSc developers: > Hi, > I have some questions about some functions of AMG and the construction time > of Jacobian matrix in the process of using. Please help me to answer them. > > 1. I see some functions about AMG in the list of PETSc functions. I wonder > if DMDA is necessary to construct vectors when using AMG precondition. > (Because the geometry described in my PDE is irregular,it contains a lot of > pipes and connections, instead of using DMDA to manage vectors, I define > discrete solution vectors by myself.) It is not used by AMG. Geometric MG either uses a DM (of which DMDA is one, but there are unstructured choices) or requires the user to call functions like PCMGSetInterpolation. AMG is probably a better choice for your problem (unless the pipe networks are such that a direct solver is practical; note that Approximate Minimum Degree ordering may be better than Nested Dissection). > 2. As far as I know, besides V-cycle, AMG method also needs auxiliary > matrix B and transfer matrix P (transfer matrix from fine mesh to coarse > mesh). Are these transformation matrices built-in or can they be provided > by users in PETSc? I'm not sure what your B represents, but see PCMGSetInterpolation to specify P. > 3. At present, in my program, I use -snes_mf_operator to solve the > non-linear PDE problem. Although I do not directly construct Jacobian > matrix, it is also expensive to construct preconditioning matrix (provided > by users). Can I control the update and decomposition time of the > preconditioning matrix, such as updating the preconditioning matrix every > two Newton steps and then ILU factorization? Can these controls be > implemented in SNESMonitor? See SNESSetLagJacobian (-snes_lag_jacobian) and SNESSetLagPreconditioner (-snes_lag_preconditioner). From mfadams at lbl.gov Sun Jul 14 10:20:26 2019 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 14 Jul 2019 11:20:26 -0400 Subject: [petsc-users] Questions about AMG and Jacobian Contruction In-Reply-To: References: Message-ID: PETSc has three AMG solvers: GAMG (native) and third party libraries hypre and ML. This probably what you want. On Sun, Jul 14, 2019 at 6:25 AM Yingjie Wu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Respected PETSc developers: > Hi, > I have some questions about some functions of AMG and the construction > time of Jacobian matrix in the process of using. Please help me to answer > them. > > 1. I see some functions about AMG in the list of PETSc functions. I wonder > if DMDA is necessary to construct vectors when using AMG precondition. > (Because the geometry described in my PDE is irregular,it contains a lot > of pipes and connections, instead of using DMDA to manage vectors, I > define discrete solution vectors by myself.) > > 2. As far as I know, besides V-cycle, AMG method also needs auxiliary > matrix B and transfer matrix P (transfer matrix from fine mesh to coarse > mesh). Are these transformation matrices built-in or can they be provided > by users in PETSc? > > 3. At present, in my program, I use -snes_mf_operator to solve the > non-linear PDE problem. Although I do not directly construct Jacobian > matrix, it is also expensive to construct preconditioning matrix (provided > by users). Can I control the update and decomposition time of the > preconditioning matrix, such as updating the preconditioning matrix every > two Newton steps and then ILU factorization? Can these controls be > implemented in SNESMonitor? > > Thanks, > Yingjie > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jul 14 11:44:51 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 14 Jul 2019 16:44:51 +0000 Subject: [petsc-users] Questions about AMG and Jacobian Contruction In-Reply-To: References: Message-ID: > On Jul 14, 2019, at 5:22 AM, Yingjie Wu via petsc-users wrote: > > Respected PETSc developers: > Hi, > I have some questions about some functions of AMG and the construction time of Jacobian matrix in the process of using. Please help me to answer them. > > 1. I see some functions about AMG in the list of PETSc functions. I wonder if DMDA is necessary to construct vectors when using AMG precondition. (Because the geometry described in my PDE is irregular,it contains a lot of pipes and connections, You might take a look at DMNetwork, this a specialized DM for network problems which it sounds like you have. Of course you don't have to use DMNetwork, you can manage the network data structure yourself. We've found that the additive Schwarz methods (PCASM) or even block Jacobi PCBJACOBI often work well for network problems, better than GAMG. GAMG doesn't know anything about the structure of networks, it is for PDEs on meshes, so there is no reason to think it would work particularly well for network problems. Barry > instead of using DMDA to manage vectors, I define discrete solution vectors by myself.) > > 2. As far as I know, besides V-cycle, AMG method also needs auxiliary matrix B and transfer matrix P (transfer matrix from fine mesh to coarse mesh). Are these transformation matrices built-in or can they be provided by users in PETSc? > > 3. At present, in my program, I use -snes_mf_operator to solve the non-linear PDE problem. Although I do not directly construct Jacobian matrix, it is also expensive to construct preconditioning matrix (provided by users). Can I control the update and decomposition time of the preconditioning matrix, such as updating the preconditioning matrix every two Newton steps and then ILU factorization? Can these controls be implemented in SNESMonitor? > > Thanks, > Yingjie From jed at jedbrown.org Sun Jul 14 12:24:26 2019 From: jed at jedbrown.org (Jed Brown) Date: Sun, 14 Jul 2019 11:24:26 -0600 Subject: [petsc-users] Questions about AMG and Jacobian Contruction In-Reply-To: References: Message-ID: <87muhgil6d.fsf@jedbrown.org> "Smith, Barry F. via petsc-users" writes: > We've found that the additive Schwarz methods (PCASM) or even block > Jacobi PCBJACOBI often work well for network problems, better than > GAMG. GAMG doesn't know anything about the structure of networks, > it is for PDEs on meshes, so there is no reason to think it would > work particularly well for network problems. The number of iterations for one-level methods (like Jacobi/block Jacobi) is roughly proportional to the diameter of your network. If your network has small diameter, those methods may be fast already. PDE meshes generally have large diameter with bounded degree; the AMG methods in PETSc are good for such problems. If you have large diameter with unbounded degree, then more specialized AMG variants such as https://arxiv.org/abs/1705.06266 can be better. From hsu at kairospower.com Mon Jul 15 18:44:37 2019 From: hsu at kairospower.com (Sean Hsu) Date: Mon, 15 Jul 2019 23:44:37 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver Message-ID: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> Hi, I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): l_max_its = 15 l_tol = 1e-8 nl_max_its = 50 nl_rel_tol = 1e-7 nl_abs_tol = 1e-9 petsc_options = '-snes_ksp_ew' petsc_options_iname = '-pc_type -snes_linesearch_type' petsc_options_value = 'lu bt' end_time = 50.0 dt = 0.5 Here is a example output from the first few timesteps of the simulation: Time Step 2, time = 0.75, dt = 0.5 0 Nonlinear |R| = 1.078014e-02 0 SNES Function norm 1.078014340559e-02 0 Linear |R| = 1.078014e-02 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 1 Linear |R| = 2.319831e-13 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 Linear solve converged due to CONVERGED_RTOL iterations 1 NEML stress update failed! Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda NEML stress update failed! Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda NEML stress update failed! Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda NEML stress update failed! Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda NEML stress update failed! Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda NEML stress update failed! Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 Solve Did NOT Converge! I really appreciate any inputs or insights, thanks for your time and help. Best, Sean [signature_16346658] Sean Hsu Mechanics of Materials Intern p 510.808.5265 e hsu at kairospower.com 707 W Tower Ave, Alameda, CA 94501 www.kairospower.com [Image result for linkedin logo] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18556 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 10320 bytes Desc: image002.png URL: From bsmith at mcs.anl.gov Mon Jul 15 18:59:25 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 15 Jul 2019 23:59:25 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver In-Reply-To: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> References: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> Message-ID: <31FF63CE-F6D7-4621-9183-EE0978B809FD@anl.gov> Function norm was 1.078014e-02 Linear solve is great, > Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda .... ... > Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda You take a microscopically small step in space and your objective function is Inf or Nan. Do you have constraints on your solution where it becomes non-physical? For example divide by an element of the solution (that would require the solution not be zero)? Is the initial solution on a constraint boundary? If you use a coarser mesh do you get this same type of behavior? If you have bounds on your solution you might need to consider solving it as a differential variational inequality (DVI) Barry > On Jul 15, 2019, at 6:44 PM, Sean Hsu via petsc-users wrote: > > Hi, > > I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): > > l_max_its = 15 > l_tol = 1e-8 > nl_max_its = 50 > nl_rel_tol = 1e-7 > nl_abs_tol = 1e-9 > petsc_options = '-snes_ksp_ew' > petsc_options_iname = '-pc_type -snes_linesearch_type' > petsc_options_value = 'lu bt' > end_time = 50.0 > dt = 0.5 > > Here is a example output from the first few timesteps of the simulation: > > Time Step 2, time = 0.75, dt = 0.5 > 0 Nonlinear |R| = 1.078014e-02 > 0 SNES Function norm 1.078014340559e-02 > 0 Linear |R| = 1.078014e-02 > 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 > 1 Linear |R| = 2.319831e-13 > 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 > Linear solve converged due to CONVERGED_RTOL iterations 1 > NEML stress update failed! > Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda > NEML stress update failed! > Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda > NEML stress update failed! > Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda > NEML stress update failed! > Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda > NEML stress update failed! > Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda > NEML stress update failed! > Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda > > Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda > > Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda > Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 > Solve Did NOT Converge! > > I really appreciate any inputs or insights, thanks for your time and help. > > Best, > > Sean > > > > Sean Hsu Mechanics of Materials Intern > p 510.808.5265 e hsu at kairospower.com > 707 W Tower Ave, Alameda, CA 94501 > www.kairospower.com From bsmith at mcs.anl.gov Mon Jul 15 19:21:05 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jul 2019 00:21:05 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver In-Reply-To: <23EDD167-EA91-47C8-B631-4BAAEA75FADA@kairospower.com> References: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> <31FF63CE-F6D7-4621-9183-EE0978B809FD@anl.gov> <23EDD167-EA91-47C8-B631-4BAAEA75FADA@kairospower.com> Message-ID: What is causing the Inf or Nan in the function 1) determine if it is inf or Nan because that can mean different things 2) determine what the physical meaning of the inf or Nan is, that is physically what is going wrong to cause it. I would back off from worrying about convergence rates a bit and instead worry about resolving this issue first because it shows something is really funky with the modeling aspect. That changing the solution by the 14th digit can cause it to go from 1.e-2 to infinite or Nan is problematic. Barry > On Jul 15, 2019, at 7:10 PM, Sean Hsu wrote: > > Hi Barry, > > Thanks for the quick response. > > This behavior shows up occasionally on coarser meshes, I don't believe I have non-physical constraints, the mesh doesn't change in the simulation and it is simply a simulation with a fix boundary on one side and a constant strain on the other side of a plate. > > I also don't have bounds on my solution. > > Usually when the line search fails, decrease time step size can usually yield converged results, but if the time step size gets too small the simulation takes a very long time to run. Please let me know if you need any more information regarding this particular issue, I will be more than happy to provide you details of my simulation. Thanks! > > Best, > > Sean > > > > > Sean Hsu Mechanics of Materials Intern > p 510.808.5265 e hsu at kairospower.com > 707 W Tower Ave, Alameda, CA 94501 > www.kairospower.com > > > > ?On 7/15/19, 5:00 PM, "Smith, Barry F." wrote: > > > Function norm was 1.078014e-02 > Linear solve is great, > >> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda .... > ... >> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda > > You take a microscopically small step in space and your objective function is Inf or Nan. > > Do you have constraints on your solution where it becomes non-physical? For example divide by an element of the solution (that would require the solution not be zero)? Is the initial solution on a constraint boundary? > > If you use a coarser mesh do you get this same type of behavior? > > If you have bounds on your solution you might need to consider solving it as a differential variational inequality (DVI) > > Barry > > > > > >> On Jul 15, 2019, at 6:44 PM, Sean Hsu via petsc-users wrote: >> >> Hi, >> >> I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): >> >> l_max_its = 15 >> l_tol = 1e-8 >> nl_max_its = 50 >> nl_rel_tol = 1e-7 >> nl_abs_tol = 1e-9 >> petsc_options = '-snes_ksp_ew' >> petsc_options_iname = '-pc_type -snes_linesearch_type' >> petsc_options_value = 'lu bt' >> end_time = 50.0 >> dt = 0.5 >> >> Here is a example output from the first few timesteps of the simulation: >> >> Time Step 2, time = 0.75, dt = 0.5 >> 0 Nonlinear |R| = 1.078014e-02 >> 0 SNES Function norm 1.078014340559e-02 >> 0 Linear |R| = 1.078014e-02 >> 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 >> 1 Linear |R| = 2.319831e-13 >> 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> NEML stress update failed! >> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda >> NEML stress update failed! >> Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda >> NEML stress update failed! >> Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda >> NEML stress update failed! >> Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda >> NEML stress update failed! >> Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda >> NEML stress update failed! >> Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda >> >> Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda >> >> Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda >> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 >> Solve Did NOT Converge! >> >> I really appreciate any inputs or insights, thanks for your time and help. >> >> Best, >> >> Sean >> >> >> >> Sean Hsu Mechanics of Materials Intern >> p 510.808.5265 e hsu at kairospower.com >> 707 W Tower Ave, Alameda, CA 94501 >> www.kairospower.com > > > From chentyl at uw.edu Mon Jul 15 22:49:57 2019 From: chentyl at uw.edu (Tyler Chen) Date: Mon, 15 Jul 2019 20:49:57 -0700 Subject: [petsc-users] Block matrix vector products? Message-ID: Hello, I am looking for a way to compute two matrix vector products simultaneously, where the matrix is the same for both products (i.e. Ax and Ay). I know some numerical libraries have the concept of "block vectors" or "basis vectors" and was wondering if there is anything similar in PETSc. Right now I am just using two MatMults sequentially, but I'm hoping there are probably better ways to do this. My matrices are MPIAIJ if that makes any difference. Best, Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 15 23:54:38 2019 From: jed at jedbrown.org (Jed Brown) Date: Mon, 15 Jul 2019 22:54:38 -0600 Subject: [petsc-users] Block matrix vector products? In-Reply-To: References: Message-ID: <87k1cipoj5.fsf@jedbrown.org> You can use MatCreateMAIJ(A,2,&M) and a single MatMult(A,xy) where xy contains the vectors x and y interlaced [x_0, y_0, x_1, y_1, ...]. There is also MatMatMult(A,X,...,&Y) where X is a MATDENSE with two columns, but I would prefer the MAIJ variant above in most cases. Tyler Chen via petsc-users writes: > Hello, > > I am looking for a way to compute two matrix vector products > simultaneously, where the matrix is the same for both products (i.e. Ax and > Ay). I know some numerical libraries have the concept of "block vectors" or > "basis vectors" and was wondering if there is anything similar in PETSc. > Right now I am just using two MatMults sequentially, but I'm hoping there > are probably better ways to do this. My matrices are MPIAIJ if that makes > any difference. > > Best, > Tyler From bsmith at mcs.anl.gov Tue Jul 16 01:45:31 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jul 2019 06:45:31 +0000 Subject: [petsc-users] Block matrix vector products? In-Reply-To: <87k1cipoj5.fsf@jedbrown.org> References: <87k1cipoj5.fsf@jedbrown.org> Message-ID: > On Jul 15, 2019, at 11:54 PM, Jed Brown via petsc-users wrote: > > You can use MatCreateMAIJ(A,2,&M) and a single MatMult(A,xy) where xy > contains the vectors x and y interlaced [x_0, y_0, x_1, y_1, ...]. > > There is also MatMatMult(A,X,...,&Y) where X is a MATDENSE with two > columns, but I would prefer the MAIJ variant above in most cases. If you are "stuck" with using non-interlaced vectors and don't want the added memcpy() of moving the vector entries from regular vectors into the dense matrix before and after the product you can use MatDenseGetArray() and VecPlaceArray() to allow the vectors to "share" the column memory of the matrix with the vectors. Using this approach the dense matrix allows one to think of the two vectors as one "block" vector. Note that though there is some speed advantage to combining the two products it is not as dramatic as one would dream about. Barry > > Tyler Chen via petsc-users writes: > >> Hello, >> >> I am looking for a way to compute two matrix vector products >> simultaneously, where the matrix is the same for both products (i.e. Ax and >> Ay). I know some numerical libraries have the concept of "block vectors" or >> "basis vectors" and was wondering if there is anything similar in PETSc. >> Right now I am just using two MatMults sequentially, but I'm hoping there >> are probably better ways to do this. My matrices are MPIAIJ if that makes >> any difference. >> >> Best, >> Tyler From jason.monschke at onscale.com Tue Jul 16 11:32:15 2019 From: jason.monschke at onscale.com (Jason Monschke) Date: Tue, 16 Jul 2019 09:32:15 -0700 Subject: [petsc-users] DMPlex, multilevel join to find face from vertices Message-ID: Hello, I have a DMPlex 3D mesh containing cells, faces, edges, and vertices. What is the best way to get the face which contains a set of vertices in this mesh? I can find the edge which contains a set of vertices using DMPlexGetJoin. However, the documentation explicitly says that it does not work for multilevel joins. My current idea is to: 1. Find the supporting edges of each vertex using DMPlexGetSupport and add them to a list of edges 2. Remove all edges that only appear once in the list 3. Use DMPlexGetJoin to find the face that is the join of these edges Does this sound like it will work? Is there a better way to do this? Cheers, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 16 17:52:33 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jul 2019 22:52:33 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver In-Reply-To: <45A38924-6C58-4109-82A4-C1E4A0FC9382@kairospower.com> References: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> <31FF63CE-F6D7-4621-9183-EE0978B809FD@anl.gov> <23EDD167-EA91-47C8-B631-4BAAEA75FADA@kairospower.com> <45A38924-6C58-4109-82A4-C1E4A0FC9382@kairospower.com> Message-ID: What time stepper are you using, something in PETSc or your own? If PETSc send -ts_view. If PETSc you can run with -ts_monitor -ts_adapt_monitor to see why it is making the tilmestep choices it is making. Unfortunately I don't think MOOSE uses PETSc time-steppers so you will need to consult with the MOOSE team on how they choose time-steps, how to monitor them and determine why it is selecting such small values. You can run a case that starts to have small time steps with -snes_monitor -snes_converged_reason and send the output. At least this will give some information on how the nonlinear solver is doing. And maybe we can make suggestions on how to proceed. Barry > On Jul 16, 2019, at 3:02 PM, Sean Hsu wrote: > > Hi Barry, > > I think one of the reasons that causes this is not prescribing enough boundary conditions in this case, resulting nan values during calculations. The linear solver issue doesn't persist anymore when I added in enough boundary conditions, as the linear solver doesn't have issue solving the system. Now my issue lies in dt dropping too low during solving as the solver failed due to timestep dropping below dtmin (2e-14). Do you have any insight into this? Again, thanks a lot for your help! > > Best, > > Sean > > > > > > Sean Hsu Mechanics of Materials Intern > p 510.808.5265 e hsu at kairospower.com > 707 W Tower Ave, Alameda, CA 94501 > www.kairospower.com > > > > ?On 7/15/19, 5:26 PM, "Smith, Barry F." wrote: > > > What is causing the Inf or Nan in the function > > 1) determine if it is inf or Nan because that can mean different things > > 2) determine what the physical meaning of the inf or Nan is, that is physically what is going wrong to cause it. > > I would back off from worrying about convergence rates a bit and instead worry about resolving this issue first because it shows something is really funky with the modeling aspect. That changing the solution by the 14th digit can cause it to go from 1.e-2 to infinite or Nan is problematic. > > > Barry > > > >> On Jul 15, 2019, at 7:10 PM, Sean Hsu wrote: >> >> Hi Barry, >> >> Thanks for the quick response. >> >> This behavior shows up occasionally on coarser meshes, I don't believe I have non-physical constraints, the mesh doesn't change in the simulation and it is simply a simulation with a fix boundary on one side and a constant strain on the other side of a plate. >> >> I also don't have bounds on my solution. >> >> Usually when the line search fails, decrease time step size can usually yield converged results, but if the time step size gets too small the simulation takes a very long time to run. Please let me know if you need any more information regarding this particular issue, I will be more than happy to provide you details of my simulation. Thanks! >> >> Best, >> >> Sean >> >> >> >> >> Sean Hsu Mechanics of Materials Intern >> p 510.808.5265 e hsu at kairospower.com >> 707 W Tower Ave, Alameda, CA 94501 >> www.kairospower.com >> >> >> >> On 7/15/19, 5:00 PM, "Smith, Barry F." wrote: >> >> >> Function norm was 1.078014e-02 >> Linear solve is great, >> >>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda .... >> ... >>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >> >> You take a microscopically small step in space and your objective function is Inf or Nan. >> >> Do you have constraints on your solution where it becomes non-physical? For example divide by an element of the solution (that would require the solution not be zero)? Is the initial solution on a constraint boundary? >> >> If you use a coarser mesh do you get this same type of behavior? >> >> If you have bounds on your solution you might need to consider solving it as a differential variational inequality (DVI) >> >> Barry >> >> >> >> >> >>> On Jul 15, 2019, at 6:44 PM, Sean Hsu via petsc-users wrote: >>> >>> Hi, >>> >>> I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): >>> >>> l_max_its = 15 >>> l_tol = 1e-8 >>> nl_max_its = 50 >>> nl_rel_tol = 1e-7 >>> nl_abs_tol = 1e-9 >>> petsc_options = '-snes_ksp_ew' >>> petsc_options_iname = '-pc_type -snes_linesearch_type' >>> petsc_options_value = 'lu bt' >>> end_time = 50.0 >>> dt = 0.5 >>> >>> Here is a example output from the first few timesteps of the simulation: >>> >>> Time Step 2, time = 0.75, dt = 0.5 >>> 0 Nonlinear |R| = 1.078014e-02 >>> 0 SNES Function norm 1.078014340559e-02 >>> 0 Linear |R| = 1.078014e-02 >>> 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 Linear |R| = 2.319831e-13 >>> 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 >>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>> NEML stress update failed! >>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda >>> NEML stress update failed! >>> Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda >>> NEML stress update failed! >>> Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda >>> NEML stress update failed! >>> Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda >>> NEML stress update failed! >>> Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda >>> NEML stress update failed! >>> Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda >>> >>> Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda >>> >>> Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda >>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 >>> Solve Did NOT Converge! >>> >>> I really appreciate any inputs or insights, thanks for your time and help. >>> >>> Best, >>> >>> Sean >>> >>> >>> >>> Sean Hsu Mechanics of Materials Intern >>> p 510.808.5265 e hsu at kairospower.com >>> 707 W Tower Ave, Alameda, CA 94501 >>> www.kairospower.com >> >> >> > > > From bsmith at mcs.anl.gov Tue Jul 16 18:21:22 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jul 2019 23:21:22 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver In-Reply-To: <610FB5C0-D323-467F-8934-F90AC0B73F2E@kairospower.com> References: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> <31FF63CE-F6D7-4621-9183-EE0978B809FD@anl.gov> <23EDD167-EA91-47C8-B631-4BAAEA75FADA@kairospower.com> <45A38924-6C58-4109-82A4-C1E4A0FC9382@kairospower.com> <610FB5C0-D323-467F-8934-F90AC0B73F2E@kairospower.com> Message-ID: <94F036E5-B748-4097-8484-2A5264884CE6@mcs.anl.gov> Ahh, run with also --ksp_monitor -ksp_converged_reason -ksp_monitor_singular_value and send the new output. For smallish problems for debugging purposes you can run with -pc_type lu to force a direct solve of the linear system and see what happens (if the linear problems are solvable at all). > On Jul 16, 2019, at 6:11 PM, Sean Hsu wrote: > > Hi Barry, > > I'm using a default time stepper from MOOSE, which is implicit-euler method. Here is the output from one of the examples where the timestep gets really small, I couldn't find a lot of information regarding DIVERGED LINEAR_SOLVE, thanks for your help: > > ??Time Step 1, time = 0.1, dt = 0.1 > 0 Nonlinear |R| = 4.012105e-03 > 0 SNES Function norm 4.012104835449e-03 > 0 Linear |R| = 4.012105e-03 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.05, dt = 0.05 > 0 Nonlinear |R| = 2.005850e-03 > 0 SNES Function norm 2.005849521314e-03 > 0 Linear |R| = 2.005850e-03 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.025, dt = 0.025 > 0 Nonlinear |R| = 1.002874e-03 > 0 SNES Function norm 1.002873988124e-03 > 0 Linear |R| = 1.002874e-03 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.0125, dt = 0.0125 > 0 Nonlinear |R| = 5.014243e-04 > 0 SNES Function norm 5.014242948706e-04 > 0 Linear |R| = 5.014243e-04 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.00625, dt = 0.00625 > 0 Nonlinear |R| = 2.507090e-04 > 0 SNES Function norm 2.507089718805e-04 > 0 Linear |R| = 2.507090e-04 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.003125, dt = 0.003125 > 0 Nonlinear |R| = 1.253537e-04 > 0 SNES Function norm 1.253536919567e-04 > 0 Linear |R| = 1.253537e-04 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.0015625, dt = 0.0015625 > 0 Nonlinear |R| = 6.267665e-05 > 0 SNES Function norm 6.267664747058e-05 > 0 Linear |R| = 6.267665e-05 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.00078125, dt = 0.00078125 > 0 Nonlinear |R| = 3.133827e-05 > 0 SNES Function norm 3.133827410694e-05 > 0 Linear |R| = 3.133827e-05 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 0.000390625, dt = 0.000390625 > 0 Nonlinear |R| = 1.566912e-05 > 0 SNES Function norm 1.566912464487e-05 > 0 Linear |R| = 1.566912e-05 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 0.000195313, dt = 0.000195313 > 0 Nonlinear |R| = 7.834559e-06 > 0 SNES Function norm 7.834559222250e-06 > 0 Linear |R| = 7.834559e-06 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 9.76563e-05, dt = 9.76563e-05 > 0 Nonlinear |R| = 3.917279e-06 > 0 SNES Function norm 3.917278835540e-06 > 0 Linear |R| = 3.917279e-06 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 4.88281e-05, dt = 4.88281e-05 > 0 Nonlinear |R| = 1.958639e-06 > 0 SNES Function norm 1.958639223338e-06 > 0 Linear |R| = 1.958639e-06 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 2.44141e-05, dt = 2.44141e-05 > 0 Nonlinear |R| = 9.793196e-07 > 0 SNES Function norm 9.793195635958e-07 > 0 Linear |R| = 9.793196e-07 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 1.2207e-05, dt = 1.2207e-05 > 0 Nonlinear |R| = 4.896598e-07 > 0 SNES Function norm 4.896597688439e-07 > 0 Linear |R| = 4.896598e-07 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 6.10352e-06, dt = 6.10352e-06 > 0 Nonlinear |R| = 2.448299e-07 > 0 SNES Function norm 2.448298810498e-07 > 0 Linear |R| = 2.448299e-07 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 3.05176e-06, dt = 3.05176e-06 > 0 Nonlinear |R| = 1.224149e-07 > 0 SNES Function norm 1.224149403502e-07 > 0 Linear |R| = 1.224149e-07 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 1.52588e-06, dt = 1.52588e-06 > 0 Nonlinear |R| = 6.120747e-08 > 0 SNES Function norm 6.120747013139e-08 > 0 Linear |R| = 6.120747e-08 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 7.62939e-07, dt = 7.62939e-07 > 0 Nonlinear |R| = 3.060373e-08 > 0 SNES Function norm 3.060373492112e-08 > 0 Linear |R| = 3.060373e-08 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 3.8147e-07, dt = 3.8147e-07 > 0 Nonlinear |R| = 1.530187e-08 > 0 SNES Function norm 1.530186849366e-08 > 0 Linear |R| = 1.530187e-08 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 1.90735e-07, dt = 1.90735e-07 > 0 Nonlinear |R| = 7.650934e-09 > 0 SNES Function norm 7.650934237796e-09 > 0 Linear |R| = 7.650934e-09 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 9.53674e-08, dt = 9.53674e-08 > 0 Nonlinear |R| = 3.825467e-09 > 0 SNES Function norm 3.825467116639e-09 > 0 Linear |R| = 3.825467e-09 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 4.76837e-08, dt = 4.76837e-08 > 0 Nonlinear |R| = 1.912734e-09 > 0 SNES Function norm 1.912733557755e-09 > 0 Linear |R| = 1.912734e-09 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 2.38419e-08, dt = 2.38419e-08 > 0 Nonlinear |R| = 9.563657e-10 > 0 SNES Function norm 9.563657094854e-10 > 0 Linear |R| = 9.563657e-10 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 1.19209e-08, dt = 1.19209e-08 > 0 Nonlinear |R| = 4.781839e-10 > 0 SNES Function norm 4.781839239582e-10 > 0 Linear |R| = 4.781839e-10 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 5.96046e-09, dt = 5.96046e-09 > 0 Nonlinear |R| = 2.390909e-10 > 0 SNES Function norm 2.390908927195e-10 > 0 Linear |R| = 2.390909e-10 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > Time Step 1, time = 2.98023e-09, dt = 2.98023e-09 > 0 Nonlinear |R| = 1.195465e-10 > 0 SNES Function norm 1.195465156083e-10 > 0 Linear |R| = 1.195465e-10 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 > Solve Did NOT Converge! > > Time Step 1, time = 1.49012e-09, dt = 1.49012e-09 > 0 Nonlinear |R| = 5.977326e-11 > 0 SNES Function norm 5.977325780361e-11 > Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 0 > Solve Converged! > Outlier Variable Residual Norms: > disp_z: 5.907986e-11 > > Best, > > Sean > > > > > > Sean Hsu Mechanics of Materials Intern > p 510.808.5265 e hsu at kairospower.com > 707 W Tower Ave, Alameda, CA 94501 > www.kairospower.com > > > > ?On 7/16/19, 3:52 PM, "Smith, Barry F." wrote: > > > What time stepper are you using, something in PETSc or your own? If PETSc send -ts_view. If PETSc you can run with -ts_monitor -ts_adapt_monitor to see why it is making the tilmestep choices it is making. > > Unfortunately I don't think MOOSE uses PETSc time-steppers so you will need to consult with the MOOSE team on how they choose time-steps, how to monitor them and determine why it is selecting such small values. > > You can run a case that starts to have small time steps with -snes_monitor -snes_converged_reason and send the output. At least this will give some information on how the nonlinear solver is doing. And maybe we can make suggestions on how to proceed. > > > Barry > > >> On Jul 16, 2019, at 3:02 PM, Sean Hsu wrote: >> >> Hi Barry, >> >> I think one of the reasons that causes this is not prescribing enough boundary conditions in this case, resulting nan values during calculations. The linear solver issue doesn't persist anymore when I added in enough boundary conditions, as the linear solver doesn't have issue solving the system. Now my issue lies in dt dropping too low during solving as the solver failed due to timestep dropping below dtmin (2e-14). Do you have any insight into this? Again, thanks a lot for your help! >> >> Best, >> >> Sean >> >> >> >> >> >> Sean Hsu Mechanics of Materials Intern >> p 510.808.5265 e hsu at kairospower.com >> 707 W Tower Ave, Alameda, CA 94501 >> www.kairospower.com >> >> >> >> On 7/15/19, 5:26 PM, "Smith, Barry F." wrote: >> >> >> What is causing the Inf or Nan in the function >> >> 1) determine if it is inf or Nan because that can mean different things >> >> 2) determine what the physical meaning of the inf or Nan is, that is physically what is going wrong to cause it. >> >> I would back off from worrying about convergence rates a bit and instead worry about resolving this issue first because it shows something is really funky with the modeling aspect. That changing the solution by the 14th digit can cause it to go from 1.e-2 to infinite or Nan is problematic. >> >> >> Barry >> >> >> >>> On Jul 15, 2019, at 7:10 PM, Sean Hsu wrote: >>> >>> Hi Barry, >>> >>> Thanks for the quick response. >>> >>> This behavior shows up occasionally on coarser meshes, I don't believe I have non-physical constraints, the mesh doesn't change in the simulation and it is simply a simulation with a fix boundary on one side and a constant strain on the other side of a plate. >>> >>> I also don't have bounds on my solution. >>> >>> Usually when the line search fails, decrease time step size can usually yield converged results, but if the time step size gets too small the simulation takes a very long time to run. Please let me know if you need any more information regarding this particular issue, I will be more than happy to provide you details of my simulation. Thanks! >>> >>> Best, >>> >>> Sean >>> >>> >>> >>> >>> Sean Hsu Mechanics of Materials Intern >>> p 510.808.5265 e hsu at kairospower.com >>> 707 W Tower Ave, Alameda, CA 94501 >>> www.kairospower.com >>> >>> >>> >>> On 7/15/19, 5:00 PM, "Smith, Barry F." wrote: >>> >>> >>> Function norm was 1.078014e-02 >>> Linear solve is great, >>> >>>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda .... >>> ... >>>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >>> >>> You take a microscopically small step in space and your objective function is Inf or Nan. >>> >>> Do you have constraints on your solution where it becomes non-physical? For example divide by an element of the solution (that would require the solution not be zero)? Is the initial solution on a constraint boundary? >>> >>> If you use a coarser mesh do you get this same type of behavior? >>> >>> If you have bounds on your solution you might need to consider solving it as a differential variational inequality (DVI) >>> >>> Barry >>> >>> >>> >>> >>> >>>> On Jul 15, 2019, at 6:44 PM, Sean Hsu via petsc-users wrote: >>>> >>>> Hi, >>>> >>>> I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): >>>> >>>> l_max_its = 15 >>>> l_tol = 1e-8 >>>> nl_max_its = 50 >>>> nl_rel_tol = 1e-7 >>>> nl_abs_tol = 1e-9 >>>> petsc_options = '-snes_ksp_ew' >>>> petsc_options_iname = '-pc_type -snes_linesearch_type' >>>> petsc_options_value = 'lu bt' >>>> end_time = 50.0 >>>> dt = 0.5 >>>> >>>> Here is a example output from the first few timesteps of the simulation: >>>> >>>> Time Step 2, time = 0.75, dt = 0.5 >>>> 0 Nonlinear |R| = 1.078014e-02 >>>> 0 SNES Function norm 1.078014340559e-02 >>>> 0 Linear |R| = 1.078014e-02 >>>> 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 >>>> 1 Linear |R| = 2.319831e-13 >>>> 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 >>>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda >>>> NEML stress update failed! >>>> Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda >>>> >>>> Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda >>>> >>>> Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda >>>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 >>>> Solve Did NOT Converge! >>>> >>>> I really appreciate any inputs or insights, thanks for your time and help. >>>> >>>> Best, >>>> >>>> Sean >>>> >>>> >>>> >>>> Sean Hsu Mechanics of Materials Intern >>>> p 510.808.5265 e hsu at kairospower.com >>>> 707 W Tower Ave, Alameda, CA 94501 >>>> www.kairospower.com >>> >>> >>> >> >> >> > > > From knepley at gmail.com Tue Jul 16 20:38:07 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Jul 2019 20:38:07 -0500 Subject: [petsc-users] DMPlex, multilevel join to find face from vertices In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 11:33 AM Jason Monschke via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I have a DMPlex 3D mesh containing cells, faces, edges, and vertices. What > is the best way to get the face which contains a set of vertices in this > mesh? I can find the edge which contains a set of vertices using > DMPlexGetJoin. However, the documentation explicitly says that it does not > work for multilevel joins. > > My current idea is to: > 1. Find the supporting edges of each vertex using DMPlexGetSupport and add > them to a list of edges > 2. Remove all edges that only appear once in the list > 3. Use DMPlexGetJoin to find the face that is the join of these edges > > Does this sound like it will work? Is there a better way to do this? > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexGetFullJoin.html I could not think of a better name. Thanks, Matt > Cheers, > Jason > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Jul 16 21:05:48 2019 From: epscodes at gmail.com (Xiangdong) Date: Tue, 16 Jul 2019 22:05:48 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver Message-ID: Hello everyone, I am new to petsc gpu and have a simple question. When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and the solve still needs some work from CPU? What are these MemCpys for during the each iteration? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 16 21:20:39 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Jul 2019 21:20:39 -0500 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 9:07 PM Xiangdong via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello everyone, > > I am new to petsc gpu and have a simple question. > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are > VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov > iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does > that mean the Krylov solve is not 100% on GPU and the solve still needs > some work from CPU? What are these MemCpys for during the each iteration? > We have GPU experts on the list, but there is definitely a communication because we do not do orthogonalization on the GPU, just the BLAS ops. This is a very small amount of data, so it just contributed latency, and I would guess that it is less then kernel launch latency. Thanks, Matt > Thank you. > > Best, > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Wed Jul 17 02:48:53 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Wed, 17 Jul 2019 16:48:53 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Hello everyone, Based on the many advices that you have provided me so far here is what I have tried. First I have a create tmp sparse matrix in CSR format based on my algorithm and then I created the MPIAIJ matrix as follows MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE, PETSC_DETERMINE, ptr, j , v, A); MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); MatSetUp(A); MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); and the RHS vector VecCreateMPI(PETSC_COMM_WORLD, this->local_size, PETSC_DETERMINE, &RHS); VecSetOption(RHS,VEC_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); VecAssemblyBegin(RHS); VecAssemblyEnd(RHS); Next, I calculated my finite volume face fluxes and stored in face-centered c-array which is accessed repeatly during matrix construction to add the contribution of different terms in PDE. After that I assembled each row using the cell-face connectivity and I set the non-zero entries for each cell which corresponds to a row using one of two approaches: (1) MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); where j_index is the global indices of cols in row "cell_global_index" and coefvalues are nnz values in that row (2) Set each row in the CSR first then dump the whole matrix into PETSC mat using MatUpdateMPIAIJWithArrays(*A,local_size, local_size, PETSC_DETERMINE,PETSC_DETERMINE , ptr, j , v ); from the master branch I did some numerical test and I found the following #1# Setting up the matrix in that way dropped the MatAssembly cost by two orders of magnitude #2# I applied similar options to the right hand side vector and got similar results as shown next This stage log is for running the routine that constructs the matrix 100 times to get consistent timings Approach (1) VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 100 1.0 1.5783e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 100 1.0 4.8351e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 AssembleMats 100 1.0 2.9211e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 callScheme 100 1.0 1.4874e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 Approach (2) VecAssemblyBegin 100 1.0 1.2069e-03 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 100 1.0 2.1696e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 200 1.0 2.0931e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 200 1.0 1.0748e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 AssembleMats 100 1.0 2.5523e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 FillCSRMat_with_MatSetValues 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 FillMat_with_MatSetValues 100 1.0 2.0326e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 6 0 0 0 0 0 callScheme 100 1.0 1.8507e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 5 0 0 0 0 0 Now as you can see the first approach is more expensive by nearly 15%, however there is the additional cost memory of of CSR in approach(2) I believe I could directly use the CSR matrix to store scheme fluxes, I don't thick it is built for repeated access since it involves a binary search cost of O(log(nnz_per_row)) Based on these results, I believe that MatSetVlaues is very expensive, so is there like a stripped down version of the function that is suitable for my application because right now it occupies more than 90% of execution time Thanks, Kamra On Sun, Jul 14, 2019 at 4:17 AM Mark Adams wrote: > MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for > its own rows, will generate an error if any process sets values for another > process. This avoids all reductions in the MatAssembly routines and thus > improves performance for very large process counts. > OK, so I am seeing the whole matrix creation, including your flux calcs > and your intermediate data structure, as taking 0.1 sec (10/100). That is > about 7% of the solve time (but this looks like it could use some > attention) or about 25 Mat-vecs (the standard work unit of an iterative > solver). > > Now I see 10% of the matrix creation time going to MatAssembly end, which > is annoying because there is no communication. > > I don't see any big problems here, except for the solver maybe, if this is > a nice friendly Laplacian. > > * I would try PETSc's Set Value routines directly. > * You might as well try > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetOption.html > > with > > MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for > its own rows, will generate an error if any process sets values for another > process. This avoids all reductions in the MatAssembly routines and thus > improves performance for very large process counts. > > This should eliminate the MatAssemblyEnd cost. > > > > On Sat, Jul 13, 2019 at 2:43 PM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Sorry about that >> I wanted to see if the assembly cost would drop with subsequent time >> steps but it was taking too long to run so I set it to solve only once >> since I was only interested in profiling the matrix builder. >> Again sorry for that >> Kamra >> >> On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: >> >>> Ok, I only see one all to KSPSolve. >>> >>> On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> >>>> This log is for 100 time-steps, not a single time step >>>> >>>> >>>> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >>>> >>>>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>>>> thing and is taking a lot of time. You should just call these once per time >>>>> step (it looks like you are just doing one time step). >>>>> >>>>> >>>>> --- Event Stage 1: Matrix Construction >>>>> >>>>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>>>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>>>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>>>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>>>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>>>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>>>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>>>> >>>>> >>>>> >>>>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>>>> petsc-users at mcs.anl.gov> wrote: >>>>> >>>>>> Hello Matt, >>>>>> Attached is the dumped entire log output using -log_view and -info. >>>>>> >>>>>> Thanks, >>>>>> Kamra >>>>>> >>>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>> >>>>>>>> Hello all, >>>>>>>> I have a few question regarding Petsc, >>>>>>>> >>>>>>> >>>>>>> Please send the entire output of a run with all the logging turned >>>>>>> on, using -log_view and -info. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Question 1: >>>>>>>> For the profiling , is it possible to only show the user defined >>>>>>>> log events in the breakdown of each stage in Log-view. >>>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>>> which should "Deactivates event logging for a PETSc object class in >>>>>>>> every stage" according to the manual. >>>>>>>> however I still see them in the stage breakdown >>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>> >>>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>>> >>>>>>>> Also is possible to clear the logs so that I can write a separate >>>>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>>>> problem and I want to know the change in performance as time goes by ) >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>> Question 2: >>>>>>>> Regarding MatSetValues >>>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>>> in "Mat A" >>>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>>> but it is very slow and it is killing my performance >>>>>>>> although the matrix was properly set using >>>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>>>> PETSC_DETERMINE, >>>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>>>> So my question is it possible to set multiple rows at once >>>>>>>> hopefully all, I checked the manual and MatSetValues can only set dense >>>>>>>> matrix block because it seems that row by row is expensive >>>>>>>> Or perhaps is it possible to copy all rows to the underlying matrix >>>>>>>> data, as I mentioned all values are local and no off-processor values ( >>>>>>>> stash is 0 ) >>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>>> mallocs. >>>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>>>> space: 0 unneeded,743028 used >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,742972 used >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>> space: 0 unneeded,743093 used >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,743036 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>> space: 0 unneeded,742938 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>> space: 0 unneeded,743049 used >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>>> space: 0 unneeded,685 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>>> space: 0 unneeded,649 used >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>>> space: 0 unneeded,1011 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>>> space: 0 unneeded,1137 used >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>>> space: 0 unneeded,658 used >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>>> space: 0 unneeded,648 used >>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>> MatSetValues() is 0 >>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>> >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>> Question 3: >>>>>>>> If all matrix and vector inserted data are local, what part of the >>>>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>>>> more time than matrix builder >>>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>>> >>>>>>>> >>>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned >>>>>>>> over six processes and it was NOT built using DM >>>>>>>> >>>>>>>> Finally the configure options are: >>>>>>>> >>>>>>>> Configure options: >>>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>>>> >>>>>>>> Sorry for such long question and thanks in advance >>>>>>>> Thanks >>>>>>>> M. Kamra >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 17 06:36:55 2019 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 17 Jul 2019 07:36:55 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: Message-ID: Also, MPI communication is done from the host, so every mat-vec will do a "CopySome" call from the device, do MPI comms, and then the next time you do GPU work it will copy from the host to get the update. On Tue, Jul 16, 2019 at 10:22 PM Matthew Knepley via petsc-users < petsc-users at mcs.anl.gov> wrote: > On Tue, Jul 16, 2019 at 9:07 PM Xiangdong via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello everyone, >> >> I am new to petsc gpu and have a simple question. >> >> When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are >> VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov >> iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does >> that mean the Krylov solve is not 100% on GPU and the solve still needs >> some work from CPU? What are these MemCpys for during the each iteration? >> > > We have GPU experts on the list, but there is definitely a communication > because we do not do orthogonalization on the GPU, > just the BLAS ops. This is a very small amount of data, so it just > contributed latency, and I would guess that it is less then kernel > launch latency. > > Thanks, > > Matt > > >> Thank you. >> >> Best, >> Xiangdong >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 17 06:42:08 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Jul 2019 06:42:08 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 2:49 AM Mohammed Mostafa wrote: > Hello everyone, > Based on the many advices that you have provided me so far here is what I > have tried. > First I have a create tmp sparse matrix in CSR format based on my > algorithm and then I created the MPIAIJ matrix as follows > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, > PETSC_DETERMINE, > PETSC_DETERMINE, ptr, j , v, A); > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > MatSetUp(A); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > and the RHS vector > VecCreateMPI(PETSC_COMM_WORLD, this->local_size, PETSC_DETERMINE, &RHS); > VecSetOption(RHS,VEC_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > Next, I calculated my finite volume face fluxes and stored in > face-centered c-array which is accessed repeatly during matrix construction > to add the contribution of different terms in PDE. > After that I assembled each row using the cell-face connectivity and I set > the non-zero entries for each cell which corresponds to a row using one of > two approaches: > (1) > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, > INSERT_VALUES); > where j_index is the global indices of cols in row "cell_global_index" and > coefvalues are nnz values in that row > > (2) > Set each row in the CSR first then dump the whole matrix into PETSC mat > using > MatUpdateMPIAIJWithArrays(*A,local_size, local_size, > PETSC_DETERMINE,PETSC_DETERMINE , ptr, j , v ); from the master branch > I did some numerical test and I found the following > #1# Setting up the matrix in that way dropped the MatAssembly cost by > two orders of magnitude > #2# I applied similar options to the right hand side vector and got > similar results as shown next > > This stage log is for running the routine that constructs the matrix 100 > times to get consistent timings > Approach (1) > VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 100 1.0 1.5783e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 100 1.0 4.8351e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > AssembleMats 100 1.0 2.9211e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > callScheme 100 1.0 1.4874e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 > > > Approach (2) > VecAssemblyBegin 100 1.0 1.2069e-03 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 100 1.0 2.1696e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 200 1.0 2.0931e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 200 1.0 1.0748e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > AssembleMats 100 1.0 2.5523e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > FillCSRMat_with_MatSetValues 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 > 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > FillMat_with_MatSetValues 100 1.0 2.0326e-01 1.1 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 1 0 0 0 0 6 0 0 0 0 0 > callScheme 100 1.0 1.8507e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 5 0 0 0 0 0 > > > Now as you can see the first approach is more expensive by nearly 15%, > however there is the additional cost memory of of CSR in approach(2) > I believe I could directly use the CSR matrix to store scheme fluxes, I > don't thick it is built for repeated access since it involves a binary > search cost of O(log(nnz_per_row)) > > Based on these results, I believe that MatSetVlaues is very expensive, so > is there like a stripped down version of the function that is suitable for > my application because right now it occupies more than 90% of execution time > 1) First, it should be easy to quickly isolate communication time. If you run on 1 process, is assembly 90% of your time? 2) How many MatVecs are you comparing against to get your 90% figure? The MatMult and KSPSolve events are not shown above, so we cannot see this 90%. Assembled matrices are only useful for solves, where you need to compute a preconditioner or for low order methods which do many MatMults. If you do only a few MatMult operations per assembly or have high order, or are not solving anything, you are better off with an unassembled application. Thanks, Matt > Thanks, > Kamra > > On Sun, Jul 14, 2019 at 4:17 AM Mark Adams wrote: > >> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for >> its own rows, will generate an error if any process sets values for another >> process. This avoids all reductions in the MatAssembly routines and thus >> improves performance for very large process counts. >> OK, so I am seeing the whole matrix creation, including your flux calcs >> and your intermediate data structure, as taking 0.1 sec (10/100). That is >> about 7% of the solve time (but this looks like it could use some >> attention) or about 25 Mat-vecs (the standard work unit of an iterative >> solver). >> >> Now I see 10% of the matrix creation time going to MatAssembly end, which >> is annoying because there is no communication. >> >> I don't see any big problems here, except for the solver maybe, if this >> is a nice friendly Laplacian. >> >> * I would try PETSc's Set Value routines directly. >> * You might as well try >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetOption.html >> >> with >> >> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values for >> its own rows, will generate an error if any process sets values for another >> process. This avoids all reductions in the MatAssembly routines and thus >> improves performance for very large process counts. >> >> This should eliminate the MatAssemblyEnd cost. >> >> >> >> On Sat, Jul 13, 2019 at 2:43 PM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> Sorry about that >>> I wanted to see if the assembly cost would drop with subsequent time >>> steps but it was taking too long to run so I set it to solve only once >>> since I was only interested in profiling the matrix builder. >>> Again sorry for that >>> Kamra >>> >>> On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: >>> >>>> Ok, I only see one all to KSPSolve. >>>> >>>> On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < >>>> mo7ammedmostafa at gmail.com> wrote: >>>> >>>>> This log is for 100 time-steps, not a single time step >>>>> >>>>> >>>>> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >>>>> >>>>>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>>>>> thing and is taking a lot of time. You should just call these once per time >>>>>> step (it looks like you are just doing one time step). >>>>>> >>>>>> >>>>>> --- Event Stage 1: Matrix Construction >>>>>> >>>>>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>>>>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>>>>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>>>>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>>>>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>>>>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>>>>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>> >>>>>>> Hello Matt, >>>>>>> Attached is the dumped entire log output using -log_view and -info. >>>>>>> >>>>>>> Thanks, >>>>>>> Kamra >>>>>>> >>>>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>> >>>>>>>>> Hello all, >>>>>>>>> I have a few question regarding Petsc, >>>>>>>>> >>>>>>>> >>>>>>>> Please send the entire output of a run with all the logging turned >>>>>>>> on, using -log_view and -info. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Question 1: >>>>>>>>> For the profiling , is it possible to only show the user defined >>>>>>>>> log events in the breakdown of each stage in Log-view. >>>>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>>>> which should "Deactivates event logging for a PETSc object class >>>>>>>>> in every stage" according to the manual. >>>>>>>>> however I still see them in the stage breakdown >>>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>>> >>>>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>>>> >>>>>>>>> Also is possible to clear the logs so that I can write a separate >>>>>>>>> profiling output file for each timestep ( since I am solving a transient >>>>>>>>> problem and I want to know the change in performance as time goes by ) >>>>>>>>> >>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>> Question 2: >>>>>>>>> Regarding MatSetValues >>>>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>>>> in "Mat A" >>>>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>>>> but it is very slow and it is killing my performance >>>>>>>>> although the matrix was properly set using >>>>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, >>>>>>>>> PETSC_DETERMINE, >>>>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur during >>>>>>>>> matsetvalues and all inserted values are local so no off-processor values >>>>>>>>> So my question is it possible to set multiple rows at once >>>>>>>>> hopefully all, I checked the manual and MatSetValues can only set dense >>>>>>>>> matrix block because it seems that row by row is expensive >>>>>>>>> Or perhaps is it possible to copy all rows to the underlying >>>>>>>>> matrix data, as I mentioned all values are local and no off-processor >>>>>>>>> values ( stash is 0 ) >>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 >>>>>>>>> mallocs. >>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>>>> mallocs. >>>>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage >>>>>>>>> space: 0 unneeded,743028 used >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>>> space: 0 unneeded,742972 used >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>>> space: 0 unneeded,743093 used >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>>> space: 0 unneeded,743036 used >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage >>>>>>>>> space: 0 unneeded,742938 used >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage >>>>>>>>> space: 0 unneeded,743049 used >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>>>> space: 0 unneeded,685 used >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>>>> space: 0 unneeded,649 used >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>>>> space: 0 unneeded,1011 used >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>>>> space: 0 unneeded,1137 used >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>>>> space: 0 unneeded,658 used >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>>>> space: 0 unneeded,648 used >>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>> MatSetValues() is 0 >>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>> >>>>>>>>> >>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>> Question 3: >>>>>>>>> If all matrix and vector inserted data are local, what part of the >>>>>>>>> vec/mat assembly consumes time because matsetvalues and matassembly consume >>>>>>>>> more time than matrix builder >>>>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>>>> >>>>>>>>> >>>>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned >>>>>>>>> over six processes and it was NOT built using DM >>>>>>>>> >>>>>>>>> Finally the configure options are: >>>>>>>>> >>>>>>>>> Configure options: >>>>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native >>>>>>>>> -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 >>>>>>>>> -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx >>>>>>>>> --with-fc=mpif90 --download-metis --download-hypre >>>>>>>>> >>>>>>>>> Sorry for such long question and thanks in advance >>>>>>>>> Thanks >>>>>>>>> M. Kamra >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Wed Jul 17 07:21:24 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Wed, 17 Jul 2019 21:21:24 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Hello, I didn't say the communication or assembly takes 90%, only matsetvalues the setting of values in the petsc matrix takes 90% of times as shown here. > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time > (sec) Flop --- Global --- --- Stage ---- > Total > Max Ratio Max > Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R > Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 100 1.0 1.5783e-04 1.5 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 100 1.0 4.8351e-04 1.4 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > AssembleMats 100 1.0 2.9211e-03 1.8 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 > 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 0 > callScheme 100 1.0 1.4874e-01 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 > 0 As you can see, no messages are exchanged, the time sink comes from copy and repeated function call since it is being called once for every row in the matrix. There is no linear solver being used in the code that I am profiling right now. For now, I am focusing on profiling construction of the Matrix and the only time sink is the overhead from MatSetValues Regards, Kamra On Wed, Jul 17, 2019 at 8:42 PM Matthew Knepley wrote: > On Wed, Jul 17, 2019 at 2:49 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Hello everyone, >> Based on the many advices that you have provided me so far here is what I >> have tried. >> First I have a create tmp sparse matrix in CSR format based on my >> algorithm and then I created the MPIAIJ matrix as follows >> >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, >> PETSC_DETERMINE, >> PETSC_DETERMINE, ptr, j , v, A); >> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); >> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >> MatSetUp(A); >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> >> and the RHS vector >> VecCreateMPI(PETSC_COMM_WORLD, this->local_size, PETSC_DETERMINE, &RHS); >> VecSetOption(RHS,VEC_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >> VecAssemblyBegin(RHS); >> VecAssemblyEnd(RHS); >> >> Next, I calculated my finite volume face fluxes and stored in >> face-centered c-array which is accessed repeatly during matrix construction >> to add the contribution of different terms in PDE. >> After that I assembled each row using the cell-face connectivity and I >> set the non-zero entries for each cell which corresponds to a row using one >> of two approaches: >> (1) >> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, >> INSERT_VALUES); >> where j_index is the global indices of cols in row "cell_global_index" >> and coefvalues are nnz values in that row >> >> (2) >> Set each row in the CSR first then dump the whole matrix into PETSC mat >> using >> MatUpdateMPIAIJWithArrays(*A,local_size, local_size, >> PETSC_DETERMINE,PETSC_DETERMINE , ptr, j , v ); from the master branch >> I did some numerical test and I found the following >> #1# Setting up the matrix in that way dropped the MatAssembly cost by >> two orders of magnitude >> #2# I applied similar options to the right hand side vector and got >> similar results as shown next >> >> This stage log is for running the routine that constructs the matrix 100 >> times to get consistent timings >> Approach (1) >> VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyEnd 100 1.0 1.5783e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 100 1.0 4.8351e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> AssembleMats 100 1.0 2.9211e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 >> callScheme 100 1.0 1.4874e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 >> >> >> Approach (2) >> VecAssemblyBegin 100 1.0 1.2069e-03 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyEnd 100 1.0 2.1696e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 200 1.0 2.0931e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 200 1.0 1.0748e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> AssembleMats 100 1.0 2.5523e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> FillCSRMat_with_MatSetValues 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 >> 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 >> FillMat_with_MatSetValues 100 1.0 2.0326e-01 1.1 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 1 0 0 0 0 6 0 0 0 0 0 >> callScheme 100 1.0 1.8507e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 1 0 0 0 0 5 0 0 0 0 0 >> >> >> Now as you can see the first approach is more expensive by nearly 15%, >> however there is the additional cost memory of of CSR in approach(2) >> I believe I could directly use the CSR matrix to store scheme fluxes, I >> don't thick it is built for repeated access since it involves a binary >> search cost of O(log(nnz_per_row)) >> >> Based on these results, I believe that MatSetVlaues is very expensive, so >> is there like a stripped down version of the function that is suitable for >> my application because right now it occupies more than 90% of execution time >> > > 1) First, it should be easy to quickly isolate communication time. If you > run on 1 process, is assembly 90% of your time? > > 2) How many MatVecs are you comparing against to get your 90% figure? The > MatMult and KSPSolve events are not shown above, > so we cannot see this 90%. > > Assembled matrices are only useful for solves, where you need to > compute a preconditioner or for low order methods which do > many MatMults. If you do only a few MatMult operations per assembly or > have high order, or are not solving anything, you are > better off with an unassembled application. > > Thanks, > > Matt > > >> Thanks, >> Kamra >> >> On Sun, Jul 14, 2019 at 4:17 AM Mark Adams wrote: >> >>> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values >>> for its own rows, will generate an error if any process sets values for >>> another process. This avoids all reductions in the MatAssembly routines and >>> thus improves performance for very large process counts. >>> OK, so I am seeing the whole matrix creation, including your flux calcs >>> and your intermediate data structure, as taking 0.1 sec (10/100). That is >>> about 7% of the solve time (but this looks like it could use some >>> attention) or about 25 Mat-vecs (the standard work unit of an iterative >>> solver). >>> >>> Now I see 10% of the matrix creation time going to MatAssembly end, >>> which is annoying because there is no communication. >>> >>> I don't see any big problems here, except for the solver maybe, if this >>> is a nice friendly Laplacian. >>> >>> * I would try PETSc's Set Value routines directly. >>> * You might as well try >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetOption.html >>> >>> with >>> >>> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values >>> for its own rows, will generate an error if any process sets values for >>> another process. This avoids all reductions in the MatAssembly routines and >>> thus improves performance for very large process counts. >>> >>> This should eliminate the MatAssemblyEnd cost. >>> >>> >>> >>> On Sat, Jul 13, 2019 at 2:43 PM Mohammed Mostafa < >>> mo7ammedmostafa at gmail.com> wrote: >>> >>>> Sorry about that >>>> I wanted to see if the assembly cost would drop with subsequent time >>>> steps but it was taking too long to run so I set it to solve only once >>>> since I was only interested in profiling the matrix builder. >>>> Again sorry for that >>>> Kamra >>>> >>>> On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: >>>> >>>>> Ok, I only see one all to KSPSolve. >>>>> >>>>> On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < >>>>> mo7ammedmostafa at gmail.com> wrote: >>>>> >>>>>> This log is for 100 time-steps, not a single time step >>>>>> >>>>>> >>>>>> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >>>>>> >>>>>>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>>>>>> thing and is taking a lot of time. You should just call these once per time >>>>>>> step (it looks like you are just doing one time step). >>>>>>> >>>>>>> >>>>>>> --- Event Stage 1: Matrix Construction >>>>>>> >>>>>>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>>>>>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>>>>>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>>>>>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>>>>>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>>>>>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>>>>>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>> >>>>>>>> Hello Matt, >>>>>>>> Attached is the dumped entire log output using -log_view and -info. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kamra >>>>>>>> >>>>>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>>> >>>>>>>>>> Hello all, >>>>>>>>>> I have a few question regarding Petsc, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Please send the entire output of a run with all the logging turned >>>>>>>>> on, using -log_view and -info. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Question 1: >>>>>>>>>> For the profiling , is it possible to only show the user defined >>>>>>>>>> log events in the breakdown of each stage in Log-view. >>>>>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>>>>> which should "Deactivates event logging for a PETSc object class >>>>>>>>>> in every stage" according to the manual. >>>>>>>>>> however I still see them in the stage breakdown >>>>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>>>> >>>>>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>>>>> >>>>>>>>>> Also is possible to clear the logs so that I can write a >>>>>>>>>> separate profiling output file for each timestep ( since I am solving a >>>>>>>>>> transient problem and I want to know the change in performance as time goes >>>>>>>>>> by ) >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>>> Question 2: >>>>>>>>>> Regarding MatSetValues >>>>>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>>>>> in "Mat A" >>>>>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>>>>> but it is very slow and it is killing my performance >>>>>>>>>> although the matrix was properly set using >>>>>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, >>>>>>>>>> this->local_size, PETSC_DETERMINE, >>>>>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur >>>>>>>>>> during matsetvalues and all inserted values are local so no off-processor >>>>>>>>>> values >>>>>>>>>> So my question is it possible to set multiple rows at once >>>>>>>>>> hopefully all, I checked the manual and MatSetValues can only set dense >>>>>>>>>> matrix block because it seems that row by row is expensive >>>>>>>>>> Or perhaps is it possible to copy all rows to the underlying >>>>>>>>>> matrix data, as I mentioned all values are local and no off-processor >>>>>>>>>> values ( stash is 0 ) >>>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>> mallocs. >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; >>>>>>>>>> storage space: 0 unneeded,743028 used >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>> storage space: 0 unneeded,742972 used >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; >>>>>>>>>> storage space: 0 unneeded,743093 used >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>> storage space: 0 unneeded,743036 used >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>> storage space: 0 unneeded,742938 used >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; >>>>>>>>>> storage space: 0 unneeded,743049 used >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>>>>> space: 0 unneeded,685 used >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>>>>> space: 0 unneeded,649 used >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>>>>> space: 0 unneeded,1011 used >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>>>>> space: 0 unneeded,1137 used >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>>>>> space: 0 unneeded,658 used >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>>>>> space: 0 unneeded,648 used >>>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>> MatSetValues() is 0 >>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>>> Question 3: >>>>>>>>>> If all matrix and vector inserted data are local, what part of >>>>>>>>>> the vec/mat assembly consumes time because matsetvalues and matassembly >>>>>>>>>> consume more time than matrix builder >>>>>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned >>>>>>>>>> over six processes and it was NOT built using DM >>>>>>>>>> >>>>>>>>>> Finally the configure options are: >>>>>>>>>> >>>>>>>>>> Configure options: >>>>>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 >>>>>>>>>> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" >>>>>>>>>> FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc >>>>>>>>>> --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre >>>>>>>>>> >>>>>>>>>> Sorry for such long question and thanks in advance >>>>>>>>>> Thanks >>>>>>>>>> M. Kamra >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 17 07:28:21 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Jul 2019 07:28:21 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 7:21 AM Mohammed Mostafa wrote: > Hello, > I didn't say the communication or assembly takes 90%, only matsetvalues > the setting of values in the petsc matrix takes 90% of times as shown here. > >> >> ------------------------------------------------------------------------------------------------------------------------ >> Event Count Time >> (sec) Flop --- Global --- --- Stage ---- >> Total >> Max Ratio Max >> Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R >> Mflop/s >> >> ------------------------------------------------------------------------------------------------------------------------ > > VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 >> 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAssemblyEnd 100 1.0 1.5783e-04 1.5 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 >> 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 100 1.0 4.8351e-04 1.4 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> AssembleMats 100 1.0 2.9211e-03 1.8 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 >> 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 0 >> callScheme 100 1.0 1.4874e-01 1.1 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 4 0 0 0 0 >> 0 > > As you can see, no messages are exchanged, the time sink comes from copy > and repeated function call since it is being called once for every row in > the matrix. > That is not clear since I cannot see what your timer is capturing. It could include computation of values, or load imbalance. > There is no linear solver being used in the code that I am profiling right > now. > For now, I am focusing on profiling construction of the Matrix and the > only time sink is the overhead from MatSetValues > There are two problems I see with the analysis. First, there is no model (even very simple) to tell you how much time you expect it to take. All proposals for optimization or bottlenecks are just guesses. Second, if there is no solve, there does not seem to be anything else going on, so how would you not expect it to take all the time. The idea would be to evaluate in the scenario in which you expect it to be used. Thanks, Matt > Regards, > Kamra > > On Wed, Jul 17, 2019 at 8:42 PM Matthew Knepley wrote: > >> On Wed, Jul 17, 2019 at 2:49 AM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> Hello everyone, >>> Based on the many advices that you have provided me so far here is what >>> I have tried. >>> First I have a create tmp sparse matrix in CSR format based on my >>> algorithm and then I created the MPIAIJ matrix as follows >>> >>> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, >>> PETSC_DETERMINE, >>> PETSC_DETERMINE, ptr, j , v, A); >>> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); >>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); >>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> MatSetUp(A); >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> >>> and the RHS vector >>> VecCreateMPI(PETSC_COMM_WORLD, this->local_size, PETSC_DETERMINE, &RHS); >>> VecSetOption(RHS,VEC_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> VecAssemblyBegin(RHS); >>> VecAssemblyEnd(RHS); >>> >>> Next, I calculated my finite volume face fluxes and stored in >>> face-centered c-array which is accessed repeatly during matrix construction >>> to add the contribution of different terms in PDE. >>> After that I assembled each row using the cell-face connectivity and I >>> set the non-zero entries for each cell which corresponds to a row using one >>> of two approaches: >>> (1) >>> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, >>> INSERT_VALUES); >>> where j_index is the global indices of cols in row "cell_global_index" >>> and coefvalues are nnz values in that row >>> >>> (2) >>> Set each row in the CSR first then dump the whole matrix into PETSC mat >>> using >>> MatUpdateMPIAIJWithArrays(*A,local_size, local_size, >>> PETSC_DETERMINE,PETSC_DETERMINE , ptr, j , v ); from the master branch >>> I did some numerical test and I found the following >>> #1# Setting up the matrix in that way dropped the MatAssembly cost by >>> two orders of magnitude >>> #2# I applied similar options to the right hand side vector and got >>> similar results as shown next >>> >>> This stage log is for running the routine that constructs the matrix 100 >>> times to get consistent timings >>> Approach (1) >>> VecAssemblyBegin 100 1.0 1.1079e-03 9.6 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyEnd 100 1.0 1.5783e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 100 1.0 1.4114e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 100 1.0 4.8351e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> AssembleMats 100 1.0 2.9211e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> FillMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 >>> 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 >>> callScheme 100 1.0 1.4874e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 0 0 0 0 4 0 0 0 0 0 >>> >>> >>> Approach (2) >>> VecAssemblyBegin 100 1.0 1.2069e-03 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyEnd 100 1.0 2.1696e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyBegin 200 1.0 2.0931e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 200 1.0 1.0748e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> AssembleMats 100 1.0 2.5523e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> FillCSRMat_with_MatSetValues 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 >>> 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 >>> FillMat_with_MatSetValues 100 1.0 2.0326e-01 1.1 0.00e+00 0.0 >>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 6 0 0 0 0 0 >>> callScheme 100 1.0 1.8507e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 0 0 0 0 5 0 0 0 0 0 >>> >>> >>> Now as you can see the first approach is more expensive by nearly 15%, >>> however there is the additional cost memory of of CSR in approach(2) >>> I believe I could directly use the CSR matrix to store scheme fluxes, I >>> don't thick it is built for repeated access since it involves a binary >>> search cost of O(log(nnz_per_row)) >>> >>> Based on these results, I believe that MatSetVlaues is very expensive, >>> so is there like a stripped down version of the function that is suitable >>> for my application because right now it occupies more than 90% of execution >>> time >>> >> >> 1) First, it should be easy to quickly isolate communication time. If you >> run on 1 process, is assembly 90% of your time? >> >> 2) How many MatVecs are you comparing against to get your 90% figure? The >> MatMult and KSPSolve events are not shown above, >> so we cannot see this 90%. >> >> Assembled matrices are only useful for solves, where you need to >> compute a preconditioner or for low order methods which do >> many MatMults. If you do only a few MatMult operations per assembly >> or have high order, or are not solving anything, you are >> better off with an unassembled application. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Kamra >>> >>> On Sun, Jul 14, 2019 at 4:17 AM Mark Adams wrote: >>> >>>> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values >>>> for its own rows, will generate an error if any process sets values for >>>> another process. This avoids all reductions in the MatAssembly routines and >>>> thus improves performance for very large process counts. >>>> OK, so I am seeing the whole matrix creation, including your flux calcs >>>> and your intermediate data structure, as taking 0.1 sec (10/100). That is >>>> about 7% of the solve time (but this looks like it could use some >>>> attention) or about 25 Mat-vecs (the standard work unit of an iterative >>>> solver). >>>> >>>> Now I see 10% of the matrix creation time going to MatAssembly end, >>>> which is annoying because there is no communication. >>>> >>>> I don't see any big problems here, except for the solver maybe, if this >>>> is a nice friendly Laplacian. >>>> >>>> * I would try PETSc's Set Value routines directly. >>>> * You might as well try >>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetOption.html >>>> >>>> with >>>> >>>> MAT_NO_OFF_PROC_ENTRIES - you know each process will only set values >>>> for its own rows, will generate an error if any process sets values for >>>> another process. This avoids all reductions in the MatAssembly routines and >>>> thus improves performance for very large process counts. >>>> >>>> This should eliminate the MatAssemblyEnd cost. >>>> >>>> >>>> >>>> On Sat, Jul 13, 2019 at 2:43 PM Mohammed Mostafa < >>>> mo7ammedmostafa at gmail.com> wrote: >>>> >>>>> Sorry about that >>>>> I wanted to see if the assembly cost would drop with subsequent time >>>>> steps but it was taking too long to run so I set it to solve only once >>>>> since I was only interested in profiling the matrix builder. >>>>> Again sorry for that >>>>> Kamra >>>>> >>>>> On Sun, Jul 14, 2019 at 3:33 AM Mark Adams wrote: >>>>> >>>>>> Ok, I only see one all to KSPSolve. >>>>>> >>>>>> On Sat, Jul 13, 2019 at 2:08 PM Mohammed Mostafa < >>>>>> mo7ammedmostafa at gmail.com> wrote: >>>>>> >>>>>>> This log is for 100 time-steps, not a single time step >>>>>>> >>>>>>> >>>>>>> On Sun, Jul 14, 2019 at 3:01 AM Mark Adams wrote: >>>>>>> >>>>>>>> You call the assembly stuff a lot (200). BuildTwoSidedF is a global >>>>>>>> thing and is taking a lot of time. You should just call these once per time >>>>>>>> step (it looks like you are just doing one time step). >>>>>>>> >>>>>>>> >>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>> >>>>>>>> BuildTwoSidedF 400 1.0 6.5222e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>>>> VecSet 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecAssemblyBegin 200 1.0 6.2633e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 >>>>>>>> VecAssemblyEnd 200 1.0 6.7163e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> VecScatterBegin 200 1.0 5.9373e-03 2.2 0.00e+00 0.0 3.6e+03 2.1e+03 0.0e+00 0 0 79 2 0 0 0 99100 0 0 >>>>>>>> VecScatterEnd 200 1.0 2.7236e-0223.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyBegin 200 1.0 3.2747e-02 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>> MatAssemblyEnd 200 1.0 9.0972e-01 1.0 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 4 0 1 0 6 9 0 1 0100 0 >>>>>>>> AssembleMats 200 1.0 1.5568e+00 1.2 0.00e+00 0.0 3.6e+03 2.1e+03 8.0e+00 6 0 79 2 6 14 0100100100 0 >>>>>>>> myMatSetValues 200 1.0 2.5367e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 25 0 0 0 0 0 >>>>>>>> setNativeMat 100 1.0 2.8223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 28 0 0 0 0 0 >>>>>>>> setNativeMatII 100 1.0 3.2174e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 31 0 0 0 0 0 >>>>>>>> callScheme 100 1.0 2.0700e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jul 12, 2019 at 11:56 PM Mohammed Mostafa via petsc-users < >>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>> >>>>>>>>> Hello Matt, >>>>>>>>> Attached is the dumped entire log output using -log_view and -info. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Kamra >>>>>>>>> >>>>>>>>> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users < >>>>>>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>>>>>> >>>>>>>>>>> Hello all, >>>>>>>>>>> I have a few question regarding Petsc, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please send the entire output of a run with all the logging >>>>>>>>>> turned on, using -log_view and -info. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Question 1: >>>>>>>>>>> For the profiling , is it possible to only show the user defined >>>>>>>>>>> log events in the breakdown of each stage in Log-view. >>>>>>>>>>> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC, >>>>>>>>>>> PetscLogEventExcludeClass(MAT_CLASSID); >>>>>>>>>>> PetscLogEventExcludeClass(VEC_CLASSID); >>>>>>>>>>> PetscLogEventExcludeClass(KSP_CLASSID); >>>>>>>>>>> PetscLogEventExcludeClass(PC_CLASSID); >>>>>>>>>>> which should "Deactivates event logging for a PETSc object class >>>>>>>>>>> in every stage" according to the manual. >>>>>>>>>>> however I still see them in the stage breakdown >>>>>>>>>>> --- Event Stage 1: Matrix Construction >>>>>>>>>>> >>>>>>>>>>> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>>>> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 >>>>>>>>>>> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 >>>>>>>>>>> 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0 >>>>>>>>>>> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 >>>>>>>>>>> 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0 >>>>>>>>>>> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 >>>>>>>>>>> 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT >>>>>>>>>>> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT >>>>>>>>>>> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT >>>>>>>>>>> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT >>>>>>>>>>> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 >>>>>>>>>>> 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT >>>>>>>>>>> >>>>>>>>>>> Also is possible to clear the logs so that I can write a >>>>>>>>>>> separate profiling output file for each timestep ( since I am solving a >>>>>>>>>>> transient problem and I want to know the change in performance as time goes >>>>>>>>>>> by ) >>>>>>>>>>> >>>>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>>>> Question 2: >>>>>>>>>>> Regarding MatSetValues >>>>>>>>>>> Right now, I writing a finite volume code, due to algorithm >>>>>>>>>>> requirement I have to write the matrix into local native format ( array of >>>>>>>>>>> arrays) and then loop through rows and use MatSetValues to set the elements >>>>>>>>>>> in "Mat A" >>>>>>>>>>> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES); >>>>>>>>>>> but it is very slow and it is killing my performance >>>>>>>>>>> although the matrix was properly set using >>>>>>>>>>> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, >>>>>>>>>>> this->local_size, PETSC_DETERMINE, >>>>>>>>>>> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A); >>>>>>>>>>> with d_nnz,and o_nnz properly assigned so no mallocs occur >>>>>>>>>>> during matsetvalues and all inserted values are local so no off-processor >>>>>>>>>>> values >>>>>>>>>>> So my question is it possible to set multiple rows at once >>>>>>>>>>> hopefully all, I checked the manual and MatSetValues can only set dense >>>>>>>>>>> matrix block because it seems that row by row is expensive >>>>>>>>>>> Or perhaps is it possible to copy all rows to the underlying >>>>>>>>>>> matrix data, as I mentioned all values are local and no off-processor >>>>>>>>>>> values ( stash is 0 ) >>>>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses >>>>>>>>>>> 0 mallocs. >>>>>>>>>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 >>>>>>>>>>> mallocs. >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; >>>>>>>>>>> storage space: 0 unneeded,743028 used >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>>> storage space: 0 unneeded,742972 used >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; >>>>>>>>>>> storage space: 0 unneeded,743093 used >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>>> storage space: 0 unneeded,743036 used >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; >>>>>>>>>>> storage space: 0 unneeded,742938 used >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; >>>>>>>>>>> storage space: 0 unneeded,743049 used >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4 >>>>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines. >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage >>>>>>>>>>> space: 0 unneeded,685 used >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage >>>>>>>>>>> space: 0 unneeded,649 used >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage >>>>>>>>>>> space: 0 unneeded,1011 used >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage >>>>>>>>>>> space: 0 unneeded,1137 used >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage >>>>>>>>>>> space: 0 unneeded,658 used >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage >>>>>>>>>>> space: 0 unneeded,648 used >>>>>>>>>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during >>>>>>>>>>> MatSetValues() is 0 >>>>>>>>>>> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 >>>>>>>>>>> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows >>>>>>>>>>> 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>>>>>>>> Question 3: >>>>>>>>>>> If all matrix and vector inserted data are local, what part of >>>>>>>>>>> the vec/mat assembly consumes time because matsetvalues and matassembly >>>>>>>>>>> consume more time than matrix builder >>>>>>>>>>> Also this is not just for the first time MAT_FINAL_ASSEMBLY >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For context the matrix in the above is nearly 1Mx1M partitioned >>>>>>>>>>> over six processes and it was NOT built using DM >>>>>>>>>>> >>>>>>>>>>> Finally the configure options are: >>>>>>>>>>> >>>>>>>>>>> Configure options: >>>>>>>>>>> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 >>>>>>>>>>> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" >>>>>>>>>>> FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc >>>>>>>>>>> --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre >>>>>>>>>>> >>>>>>>>>>> Sorry for such long question and thanks in advance >>>>>>>>>>> Thanks >>>>>>>>>>> M. Kamra >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Wed Jul 17 08:50:59 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Wed, 17 Jul 2019 22:50:59 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Sorry for the confusion First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role. So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 0 FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 I used the same code but simply filled a different Matrix something like for ( i =0; i < nRows;i++) { // ....... // Some code to get j_index, coefvalues // Method1 MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); //Method2 for ( int k = 0;k < nnz_per_row; k++) EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; } Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2. I found the cost to be trivial in comparison to when either one of the methods is used. I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice. Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ? Maybe some setting or MatOption I should be using so far this what I have been using MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE, PETSC_DETERMINE, ptr, j , v, A); MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); MatSetUp(A); MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); Thanks, Kamra -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 17 09:58:54 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Jul 2019 09:58:54 -0500 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa wrote: > Sorry for the confusion > First I fully acknowledge that setting Matrix non-zeros or copying in > general is not cheap and memory access pattern can play an important role. > So to establish a baseline to compare with, I tried setting the same > matrix but in an Eigen Sparse Matrix and the timings are as follows > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 > 0 > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 > Great. This helps. Two things would help me narrow down what is happening. 1) Are you timing only the insertion of values, or computation and insertion? 2) Can you tell me how many values are inserted? Thanks, Matt > I used the same code but simply filled a different Matrix something like > > for ( i =0; i < nRows;i++) > { > // > ....... > // Some code to get j_index, coefvalues > // Method1 > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, > INSERT_VALUES); > > //Method2 > for ( int k = 0;k < nnz_per_row; k++) > EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; > > } > Please note that only one of the two methods is being used at a time. > Also, I separately time the code section used to < j_index, coefvalues> > but simpling disabling both Method1 and Method2. > I found the cost to be trivial in comparison to when either one of the > methods is used. > I used Eigen out of convenience since I used for some vector and tensor > arithmetics somewhere else in the code and it may not be the best choice. > Since in PetscMatrix we technically fill two matrices: diagonal and > off-diagonal so I expected some difference but is that normal or am I > missing something. ? > Maybe some setting or MatOption I should be using so far this what I have > been using > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, > PETSC_DETERMINE, > PETSC_DETERMINE, ptr, j , v, A); > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > MatSetUp(A); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > Thanks, > Kamra > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Wed Jul 17 10:15:22 2019 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 17 Jul 2019 15:15:22 +0000 Subject: [petsc-users] Optimizing solver and consistent converging on non-linear solver In-Reply-To: <94F036E5-B748-4097-8484-2A5264884CE6@mcs.anl.gov> References: <4CCDB87D-B55D-46BD-BB1B-3D22F281553C@kairospower.com> <31FF63CE-F6D7-4621-9183-EE0978B809FD@anl.gov> <23EDD167-EA91-47C8-B631-4BAAEA75FADA@kairospower.com> <45A38924-6C58-4109-82A4-C1E4A0FC9382@kairospower.com> <610FB5C0-D323-467F-8934-F90AC0B73F2E@kairospower.com> <94F036E5-B748-4097-8484-2A5264884CE6@mcs.anl.gov> Message-ID: -pc_type lu has already been used in the PETSc options. The problem could be caused by a bad Jacobian. I would suggest to run -snes_test_jacobian for a small case. Hong (Mr.) > On Jul 16, 2019, at 6:21 PM, Smith, Barry F. via petsc-users wrote: > > > Ahh, run with also --ksp_monitor -ksp_converged_reason -ksp_monitor_singular_value and send the new output. > > > For smallish problems for debugging purposes you can run with -pc_type lu to force a direct solve of the linear system and see what happens (if the linear problems are solvable at all). > > >> On Jul 16, 2019, at 6:11 PM, Sean Hsu wrote: >> >> Hi Barry, >> >> I'm using a default time stepper from MOOSE, which is implicit-euler method. Here is the output from one of the examples where the timestep gets really small, I couldn't find a lot of information regarding DIVERGED LINEAR_SOLVE, thanks for your help: >> >> ??Time Step 1, time = 0.1, dt = 0.1 >> 0 Nonlinear |R| = 4.012105e-03 >> 0 SNES Function norm 4.012104835449e-03 >> 0 Linear |R| = 4.012105e-03 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.05, dt = 0.05 >> 0 Nonlinear |R| = 2.005850e-03 >> 0 SNES Function norm 2.005849521314e-03 >> 0 Linear |R| = 2.005850e-03 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.025, dt = 0.025 >> 0 Nonlinear |R| = 1.002874e-03 >> 0 SNES Function norm 1.002873988124e-03 >> 0 Linear |R| = 1.002874e-03 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.0125, dt = 0.0125 >> 0 Nonlinear |R| = 5.014243e-04 >> 0 SNES Function norm 5.014242948706e-04 >> 0 Linear |R| = 5.014243e-04 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.00625, dt = 0.00625 >> 0 Nonlinear |R| = 2.507090e-04 >> 0 SNES Function norm 2.507089718805e-04 >> 0 Linear |R| = 2.507090e-04 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.003125, dt = 0.003125 >> 0 Nonlinear |R| = 1.253537e-04 >> 0 SNES Function norm 1.253536919567e-04 >> 0 Linear |R| = 1.253537e-04 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.0015625, dt = 0.0015625 >> 0 Nonlinear |R| = 6.267665e-05 >> 0 SNES Function norm 6.267664747058e-05 >> 0 Linear |R| = 6.267665e-05 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.00078125, dt = 0.00078125 >> 0 Nonlinear |R| = 3.133827e-05 >> 0 SNES Function norm 3.133827410694e-05 >> 0 Linear |R| = 3.133827e-05 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 0.000390625, dt = 0.000390625 >> 0 Nonlinear |R| = 1.566912e-05 >> 0 SNES Function norm 1.566912464487e-05 >> 0 Linear |R| = 1.566912e-05 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 0.000195313, dt = 0.000195313 >> 0 Nonlinear |R| = 7.834559e-06 >> 0 SNES Function norm 7.834559222250e-06 >> 0 Linear |R| = 7.834559e-06 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 9.76563e-05, dt = 9.76563e-05 >> 0 Nonlinear |R| = 3.917279e-06 >> 0 SNES Function norm 3.917278835540e-06 >> 0 Linear |R| = 3.917279e-06 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 4.88281e-05, dt = 4.88281e-05 >> 0 Nonlinear |R| = 1.958639e-06 >> 0 SNES Function norm 1.958639223338e-06 >> 0 Linear |R| = 1.958639e-06 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 2.44141e-05, dt = 2.44141e-05 >> 0 Nonlinear |R| = 9.793196e-07 >> 0 SNES Function norm 9.793195635958e-07 >> 0 Linear |R| = 9.793196e-07 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 1.2207e-05, dt = 1.2207e-05 >> 0 Nonlinear |R| = 4.896598e-07 >> 0 SNES Function norm 4.896597688439e-07 >> 0 Linear |R| = 4.896598e-07 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 6.10352e-06, dt = 6.10352e-06 >> 0 Nonlinear |R| = 2.448299e-07 >> 0 SNES Function norm 2.448298810498e-07 >> 0 Linear |R| = 2.448299e-07 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 3.05176e-06, dt = 3.05176e-06 >> 0 Nonlinear |R| = 1.224149e-07 >> 0 SNES Function norm 1.224149403502e-07 >> 0 Linear |R| = 1.224149e-07 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 1.52588e-06, dt = 1.52588e-06 >> 0 Nonlinear |R| = 6.120747e-08 >> 0 SNES Function norm 6.120747013139e-08 >> 0 Linear |R| = 6.120747e-08 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 7.62939e-07, dt = 7.62939e-07 >> 0 Nonlinear |R| = 3.060373e-08 >> 0 SNES Function norm 3.060373492112e-08 >> 0 Linear |R| = 3.060373e-08 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 3.8147e-07, dt = 3.8147e-07 >> 0 Nonlinear |R| = 1.530187e-08 >> 0 SNES Function norm 1.530186849366e-08 >> 0 Linear |R| = 1.530187e-08 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 1.90735e-07, dt = 1.90735e-07 >> 0 Nonlinear |R| = 7.650934e-09 >> 0 SNES Function norm 7.650934237796e-09 >> 0 Linear |R| = 7.650934e-09 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 9.53674e-08, dt = 9.53674e-08 >> 0 Nonlinear |R| = 3.825467e-09 >> 0 SNES Function norm 3.825467116639e-09 >> 0 Linear |R| = 3.825467e-09 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 4.76837e-08, dt = 4.76837e-08 >> 0 Nonlinear |R| = 1.912734e-09 >> 0 SNES Function norm 1.912733557755e-09 >> 0 Linear |R| = 1.912734e-09 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 2.38419e-08, dt = 2.38419e-08 >> 0 Nonlinear |R| = 9.563657e-10 >> 0 SNES Function norm 9.563657094854e-10 >> 0 Linear |R| = 9.563657e-10 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 1.19209e-08, dt = 1.19209e-08 >> 0 Nonlinear |R| = 4.781839e-10 >> 0 SNES Function norm 4.781839239582e-10 >> 0 Linear |R| = 4.781839e-10 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 5.96046e-09, dt = 5.96046e-09 >> 0 Nonlinear |R| = 2.390909e-10 >> 0 SNES Function norm 2.390908927195e-10 >> 0 Linear |R| = 2.390909e-10 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> Time Step 1, time = 2.98023e-09, dt = 2.98023e-09 >> 0 Nonlinear |R| = 1.195465e-10 >> 0 SNES Function norm 1.195465156083e-10 >> 0 Linear |R| = 1.195465e-10 >> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0 >> Solve Did NOT Converge! >> >> Time Step 1, time = 1.49012e-09, dt = 1.49012e-09 >> 0 Nonlinear |R| = 5.977326e-11 >> 0 SNES Function norm 5.977325780361e-11 >> Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 0 >> Solve Converged! >> Outlier Variable Residual Norms: >> disp_z: 5.907986e-11 >> >> Best, >> >> Sean >> >> >> >> >> >> Sean Hsu Mechanics of Materials Intern >> p 510.808.5265 e hsu at kairospower.com >> 707 W Tower Ave, Alameda, CA 94501 >> www.kairospower.com >> >> >> >> ?On 7/16/19, 3:52 PM, "Smith, Barry F." wrote: >> >> >> What time stepper are you using, something in PETSc or your own? If PETSc send -ts_view. If PETSc you can run with -ts_monitor -ts_adapt_monitor to see why it is making the tilmestep choices it is making. >> >> Unfortunately I don't think MOOSE uses PETSc time-steppers so you will need to consult with the MOOSE team on how they choose time-steps, how to monitor them and determine why it is selecting such small values. >> >> You can run a case that starts to have small time steps with -snes_monitor -snes_converged_reason and send the output. At least this will give some information on how the nonlinear solver is doing. And maybe we can make suggestions on how to proceed. >> >> >> Barry >> >> >>> On Jul 16, 2019, at 3:02 PM, Sean Hsu wrote: >>> >>> Hi Barry, >>> >>> I think one of the reasons that causes this is not prescribing enough boundary conditions in this case, resulting nan values during calculations. The linear solver issue doesn't persist anymore when I added in enough boundary conditions, as the linear solver doesn't have issue solving the system. Now my issue lies in dt dropping too low during solving as the solver failed due to timestep dropping below dtmin (2e-14). Do you have any insight into this? Again, thanks a lot for your help! >>> >>> Best, >>> >>> Sean >>> >>> >>> >>> >>> >>> Sean Hsu Mechanics of Materials Intern >>> p 510.808.5265 e hsu at kairospower.com >>> 707 W Tower Ave, Alameda, CA 94501 >>> www.kairospower.com >>> >>> >>> >>> On 7/15/19, 5:26 PM, "Smith, Barry F." wrote: >>> >>> >>> What is causing the Inf or Nan in the function >>> >>> 1) determine if it is inf or Nan because that can mean different things >>> >>> 2) determine what the physical meaning of the inf or Nan is, that is physically what is going wrong to cause it. >>> >>> I would back off from worrying about convergence rates a bit and instead worry about resolving this issue first because it shows something is really funky with the modeling aspect. That changing the solution by the 14th digit can cause it to go from 1.e-2 to infinite or Nan is problematic. >>> >>> >>> Barry >>> >>> >>> >>>> On Jul 15, 2019, at 7:10 PM, Sean Hsu wrote: >>>> >>>> Hi Barry, >>>> >>>> Thanks for the quick response. >>>> >>>> This behavior shows up occasionally on coarser meshes, I don't believe I have non-physical constraints, the mesh doesn't change in the simulation and it is simply a simulation with a fix boundary on one side and a constant strain on the other side of a plate. >>>> >>>> I also don't have bounds on my solution. >>>> >>>> Usually when the line search fails, decrease time step size can usually yield converged results, but if the time step size gets too small the simulation takes a very long time to run. Please let me know if you need any more information regarding this particular issue, I will be more than happy to provide you details of my simulation. Thanks! >>>> >>>> Best, >>>> >>>> Sean >>>> >>>> >>>> >>>> >>>> Sean Hsu Mechanics of Materials Intern >>>> p 510.808.5265 e hsu at kairospower.com >>>> 707 W Tower Ave, Alameda, CA 94501 >>>> www.kairospower.com >>>> >>>> >>>> >>>> On 7/15/19, 5:00 PM, "Smith, Barry F." wrote: >>>> >>>> >>>> Function norm was 1.078014e-02 >>>> Linear solve is great, >>>> >>>>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda .... >>>> ... >>>>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >>>> >>>> You take a microscopically small step in space and your objective function is Inf or Nan. >>>> >>>> Do you have constraints on your solution where it becomes non-physical? For example divide by an element of the solution (that would require the solution not be zero)? Is the initial solution on a constraint boundary? >>>> >>>> If you use a coarser mesh do you get this same type of behavior? >>>> >>>> If you have bounds on your solution you might need to consider solving it as a differential variational inequality (DVI) >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>> >>>>> On Jul 15, 2019, at 6:44 PM, Sean Hsu via petsc-users wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have been using blackbear (a moose framework app) to do a simple tensile test simulation on a complex material model with high amount of elements. I was able to get consistent and quick solution results with low amount of elements, however once the amount of element start increasing, the solver won?t converge consistently and dt usually drops to a very low value, causing the simulation to run for a very long time. I am seeking recommendations for tips to optimize the solver so I can get consistent and faster convergence rate. Here is the petsc option that I am using (along with SMP preconditioner): >>>>> >>>>> l_max_its = 15 >>>>> l_tol = 1e-8 >>>>> nl_max_its = 50 >>>>> nl_rel_tol = 1e-7 >>>>> nl_abs_tol = 1e-9 >>>>> petsc_options = '-snes_ksp_ew' >>>>> petsc_options_iname = '-pc_type -snes_linesearch_type' >>>>> petsc_options_value = 'lu bt' >>>>> end_time = 50.0 >>>>> dt = 0.5 >>>>> >>>>> Here is a example output from the first few timesteps of the simulation: >>>>> >>>>> Time Step 2, time = 0.75, dt = 0.5 >>>>> 0 Nonlinear |R| = 1.078014e-02 >>>>> 0 SNES Function norm 1.078014340559e-02 >>>>> 0 Linear |R| = 1.078014e-02 >>>>> 0 KSP unpreconditioned resid norm 1.078014340559e-02 true resid norm 1.078014340559e-02 ||r(i)||/||b|| 1.000000000000e+00 >>>>> 1 Linear |R| = 2.319831e-13 >>>>> 1 KSP unpreconditioned resid norm 2.319831277078e-13 true resid norm 2.255163534674e-13 ||r(i)||/||b|| 2.091960607412e-11 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 1. is Inf or Nan, cutting lambda >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 0.5 is Inf or Nan, cutting lambda >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 0.25 is Inf or Nan, cutting lambda >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 0.125 is Inf or Nan, cutting lambda >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 0.0625 is Inf or Nan, cutting lambda >>>>> NEML stress update failed! >>>>> Line search: objective function at lambdas = 0.03125 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.015625 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.0078125 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.00390625 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.00195312 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.000976562 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.000488281 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.000244141 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 0.00012207 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 6.10352e-05 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 3.05176e-05 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.52588e-05 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 7.62939e-06 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 3.8147e-06 is Inf or Nan, cutting lambda >>>>> >>>>> Line search: objective function at lambdas = 1.90735e-06 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 9.53674e-07 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 4.76837e-07 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 2.38419e-07 is Inf or Nan, cutting lambda >>>>> >>>>> Line search: objective function at lambdas = 1.19209e-07 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 5.96046e-08 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 2.98023e-08 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.49012e-08 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 7.45058e-09 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 3.72529e-09 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.86265e-09 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 9.31323e-10 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 4.65661e-10 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 2.32831e-10 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.16415e-10 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 5.82077e-11 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 2.91038e-11 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.45519e-11 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 7.27596e-12 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 3.63798e-12 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 1.81899e-12 is Inf or Nan, cutting lambda >>>>> Line search: objective function at lambdas = 9.09495e-13 is Inf or Nan, cutting lambda >>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0 >>>>> Solve Did NOT Converge! >>>>> >>>>> I really appreciate any inputs or insights, thanks for your time and help. >>>>> >>>>> Best, >>>>> >>>>> Sean >>>>> >>>>> >>>>> >>>>> Sean Hsu Mechanics of Materials Intern >>>>> p 510.808.5265 e hsu at kairospower.com >>>>> 707 W Tower Ave, Alameda, CA 94501 >>>>> www.kairospower.com >>>> >>>> >>>> >>> >>> >>> >> >> >> > From dave.mayhem23 at gmail.com Wed Jul 17 10:24:26 2019 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 17 Jul 2019 16:24:26 +0100 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Can you please send your code? It would greatly help Matt (and others looking at this thread) to understand the performance you see and to offer advice on how (if possible) to improve it. On Wed, 17 Jul 2019 at 16:00, Matthew Knepley via petsc-users < petsc-users at mcs.anl.gov> wrote: > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Sorry for the confusion >> First I fully acknowledge that setting Matrix non-zeros or copying in >> general is not cheap and memory access pattern can play an important role. >> So to establish a baseline to compare with, I tried setting the same >> matrix but in an Eigen Sparse Matrix and the timings are as follows >> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 >> 0 >> FilEigenMat 100 1.0 >> 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 >> 0 0 0 0 >> > > Great. This helps. Two things would help me narrow down what is happening. > > 1) Are you timing only the insertion of values, or computation and > insertion? > > 2) Can you tell me how many values are inserted? > > Thanks, > > Matt > > >> I used the same code but simply filled a different Matrix something like >> >> for ( i =0; i < nRows;i++) >> { >> // >> ....... >> // Some code to get j_index, coefvalues >> // Method1 >> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, >> INSERT_VALUES); >> >> //Method2 >> for ( int k = 0;k < nnz_per_row; k++) >> EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; >> >> } >> Please note that only one of the two methods is being used at a time. >> Also, I separately time the code section used to < j_index, coefvalues> >> but simpling disabling both Method1 and Method2. >> I found the cost to be trivial in comparison to when either one of the >> methods is used. >> I used Eigen out of convenience since I used for some vector and tensor >> arithmetics somewhere else in the code and it may not be the best choice. >> Since in PetscMatrix we technically fill two matrices: diagonal and >> off-diagonal so I expected some difference but is that normal or am I >> missing something. ? >> Maybe some setting or MatOption I should be using so far this what I have >> been using >> >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, >> PETSC_DETERMINE, >> PETSC_DETERMINE, ptr, j , v, A); >> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); >> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >> MatSetUp(A); >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> >> Thanks, >> Kamra >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Wed Jul 17 10:34:30 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Thu, 18 Jul 2019 00:34:30 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Regarding the first point > > 1) Are you timing only the insertion of values, or computation and > insertion? I am timing both, the computation and insertion of values but as I said I timed three scenarios 1-computation only and no insertion Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0 2-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 > 0 > 3-computation and insertion into eigen matrix > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 I timed 100 times to get a reasonably accurate timings as for the second point > 2) Can you tell me how many values are inserted? For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376) In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row the number of off-diagonal onnz in total is 648 nnz So I insert nearly 4 values 186062 times ~= 744248 times per mpi process Thanks, Kamra On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley wrote: > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > >> Sorry for the confusion >> First I fully acknowledge that setting Matrix non-zeros or copying in >> general is not cheap and memory access pattern can play an important role. >> So to establish a baseline to compare with, I tried setting the same >> matrix but in an Eigen Sparse Matrix and the timings are as follows >> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 >> 0 >> FilEigenMat 100 1.0 >> 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 >> 0 0 0 0 >> > > Great. This helps. Two things would help me narrow down what is happening. > > 1) Are you timing only the insertion of values, or computation and > insertion? > > 2) Can you tell me how many values are inserted? > > Thanks, > > Matt > > >> I used the same code but simply filled a different Matrix something like >> >> for ( i =0; i < nRows;i++) >> { >> // >> ....... >> // Some code to get j_index, coefvalues >> // Method1 >> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, >> INSERT_VALUES); >> >> //Method2 >> for ( int k = 0;k < nnz_per_row; k++) >> EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; >> >> } >> Please note that only one of the two methods is being used at a time. >> Also, I separately time the code section used to < j_index, coefvalues> >> but simpling disabling both Method1 and Method2. >> I found the cost to be trivial in comparison to when either one of the >> methods is used. >> I used Eigen out of convenience since I used for some vector and tensor >> arithmetics somewhere else in the code and it may not be the best choice. >> Since in PetscMatrix we technically fill two matrices: diagonal and >> off-diagonal so I expected some difference but is that normal or am I >> missing something. ? >> Maybe some setting or MatOption I should be using so far this what I have >> been using >> >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, >> PETSC_DETERMINE, >> PETSC_DETERMINE, ptr, j , v, A); >> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); >> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); >> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >> MatSetUp(A); >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> >> Thanks, >> Kamra >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From KJiao at slb.com Wed Jul 17 11:27:02 2019 From: KJiao at slb.com (Kun Jiao) Date: Wed, 17 Jul 2019 16:27:02 +0000 Subject: [petsc-users] MatMultTranspose memory usage Message-ID: Hi, I was using MatMultTranpose and MatMult to solver a linear system. However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. This basically double my memory requirement to solve my system. I remember mkl's routine can do inplace matrix transpose vector product, without transposing the matrix itself. Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. Any help is greatly appreciated. Regards, Kun Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From hong at aspiritech.org Wed Jul 17 12:03:09 2019 From: hong at aspiritech.org (hong at aspiritech.org) Date: Wed, 17 Jul 2019 12:03:09 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: Kun: How do you know 'MatMultTranpose creates an extra memory copy of matrix'? Hong Hi, > > > > I was using MatMultTranpose and MatMult to solver a linear system. > > > > However we found out, MatMultTranpose create an extra memory copy of > matrix for its operation. This extra memory copy is not stated everywhere > in petsc manual. > > > > This basically double my memory requirement to solve my system. > > > > I remember mkl?s routine can do inplace matrix transpose vector product, > without transposing the matrix itself. > > > > Is this always the case? Or there is way to make petsc to do inplace > matrix transpose vector product. > > > > Any help is greatly appreciated. > > > > Regards, > > Kun > > > > Schlumberger-Private > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Wed Jul 17 13:52:32 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Wed, 17 Jul 2019 13:52:32 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < petsc-users at mcs.anl.gov> wrote: > Kun: > How do you know 'MatMultTranpose creates an extra memory copy of matrix'? > Hong > > Hi, >> >> >> >> I was using MatMultTranpose and MatMult to solver a linear system. >> >> >> >> However we found out, MatMultTranpose create an extra memory copy of >> matrix for its operation. This extra memory copy is not stated everywhere >> in petsc manual. >> >> >> >> This basically double my memory requirement to solve my system. >> >> >> >> I remember mkl?s routine can do inplace matrix transpose vector product, >> without transposing the matrix itself. >> >> >> >> Is this always the case? Or there is way to make petsc to do inplace >> matrix transpose vector product. >> >> >> >> Any help is greatly appreciated. >> >> >> >> Regards, >> >> Kun >> >> >> >> Schlumberger-Private >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Wed Jul 17 15:33:48 2019 From: epscodes at gmail.com (Xiangdong) Date: Wed, 17 Jul 2019 16:33:48 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: Message-ID: Thanks for your explanation. Is it correct to say that ONLY the VecNorm, VecDot, VecMDot, MatMult operations in GMRES are done on GPU and everything else is done on CPU? Thanks. Xiangdong On Wed, Jul 17, 2019 at 7:34 AM Mark Adams wrote: > Also, MPI communication is done from the host, so every mat-vec will do a > "CopySome" call from the device, do MPI comms, and then the next time you > do GPU work it will copy from the host to get the update. > > On Tue, Jul 16, 2019 at 10:22 PM Matthew Knepley via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> On Tue, Jul 16, 2019 at 9:07 PM Xiangdong via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hello everyone, >>> >>> I am new to petsc gpu and have a simple question. >>> >>> When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are >>> VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov >>> iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does >>> that mean the Krylov solve is not 100% on GPU and the solve still needs >>> some work from CPU? What are these MemCpys for during the each iteration? >>> >> >> We have GPU experts on the list, but there is definitely a communication >> because we do not do orthogonalization on the GPU, >> just the BLAS ops. This is a very small amount of data, so it just >> contributed latency, and I would guess that it is less then kernel >> launch latency. >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> >>> Best, >>> Xiangdong >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 17 15:45:07 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Jul 2019 15:45:07 -0500 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 3:33 PM Xiangdong wrote: > Thanks for your explanation. > > Is it correct to say that ONLY the VecNorm, VecDot, VecMDot, MatMult > operations in GMRES are done on GPU and everything else is done on CPU? > You are missing VecAXPY(). If instead you list what is not done, I think it is only the duiagonalization of H by Givens rotations, but this would be latency dominated on a GPU and probably slower than the CPU. Thanks, Matt > Thanks. > > Xiangdong > > On Wed, Jul 17, 2019 at 7:34 AM Mark Adams wrote: > >> Also, MPI communication is done from the host, so every mat-vec will do a >> "CopySome" call from the device, do MPI comms, and then the next time you >> do GPU work it will copy from the host to get the update. >> >> On Tue, Jul 16, 2019 at 10:22 PM Matthew Knepley via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> On Tue, Jul 16, 2019 at 9:07 PM Xiangdong via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello everyone, >>>> >>>> I am new to petsc gpu and have a simple question. >>>> >>>> When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are >>>> VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov >>>> iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does >>>> that mean the Krylov solve is not 100% on GPU and the solve still needs >>>> some work from CPU? What are these MemCpys for during the each iteration? >>>> >>> >>> We have GPU experts on the list, but there is definitely a communication >>> because we do not do orthogonalization on the GPU, >>> just the BLAS ops. This is a very small amount of data, so it just >>> contributed latency, and I would guess that it is less then kernel >>> launch latency. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you. >>>> >>>> Best, >>>> Xiangdong >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Jul 17 16:55:54 2019 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Wed, 17 Jul 2019 21:55:54 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: Karl: What matrix format do you use? Run it in parallel or sequential? Hong We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users > wrote: Kun: How do you know 'MatMultTranpose creates an extra memory copy of matrix'? Hong Hi, I was using MatMultTranpose and MatMult to solver a linear system. However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. This basically double my memory requirement to solve my system. I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. Any help is greatly appreciated. Regards, Kun Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Wed Jul 17 17:26:30 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Wed, 17 Jul 2019 17:26:30 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: MatCreateMPIAIJMKL parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: > Karl: > What matrix format do you use? Run it in parallel or sequential? > Hong > > We used /proc/self/stat to track the resident set size during program run, >> and we saw the resident set size jumped by the size of the matrix right >> after we did matmulttranspose. >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Kun: >>> How do you know 'MatMultTranpose creates an extra memory copy of matrix'? >>> Hong >>> >>> Hi, >>>> >>>> >>>> >>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>> >>>> >>>> >>>> However we found out, MatMultTranpose create an extra memory copy of >>>> matrix for its operation. This extra memory copy is not stated everywhere >>>> in petsc manual. >>>> >>>> >>>> >>>> This basically double my memory requirement to solve my system. >>>> >>>> >>>> >>>> I remember mkl?s routine can do inplace matrix transpose vector >>>> product, without transposing the matrix itself. >>>> >>>> >>>> >>>> Is this always the case? Or there is way to make petsc to do inplace >>>> matrix transpose vector product. >>>> >>>> >>>> >>>> Any help is greatly appreciated. >>>> >>>> >>>> >>>> Regards, >>>> >>>> Kun >>>> >>>> >>>> >>>> Schlumberger-Private >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Wed Jul 17 17:31:20 2019 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 18 Jul 2019 10:31:20 +1200 Subject: [petsc-users] Copying to Vec with boundary cells Message-ID: hi I have a Vec created on a distributed DMPlex (using DMCreateGlobalVector() or DMCreateLocalVector()). I then add FV boundary condition cells to the DMPlex, using DMPlexConstructGhostCells(). I can then create a new Vec on the new DM. What is the best way to copy the values from the old Vec to the new one? Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 From a.croucher at auckland.ac.nz Wed Jul 17 18:47:55 2019 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 18 Jul 2019 11:47:55 +1200 Subject: [petsc-users] Copying to Vec with boundary cells In-Reply-To: References: Message-ID: <13fbec29-97b5-8e04-6fd2-990132a89d31@auckland.ac.nz> Actually it looks like a simple VecCopy() works in this situation. It checks the local sizes of the two vectors, and decides they are still the same, so is happy to do the copy. - Adrian On 18/07/19 10:31 AM, Adrian Croucher wrote: > hi > > I have a Vec created on a distributed DMPlex (using > DMCreateGlobalVector() or DMCreateLocalVector()). > > I then add FV boundary condition cells to the DMPlex, using > DMPlexConstructGhostCells(). > > I can then create a new Vec on the new DM. What is the best way to > copy the values from the old Vec to the new one? > > Regards, Adrian > -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 4611 From karl.linkui at gmail.com Wed Jul 17 20:46:16 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Wed, 17 Jul 2019 20:46:16 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: > MatCreateMPIAIJMKL > > parallel and sequential exhibit the same behavior. In fact, we found that > doing matmult will increase the memory by the size of matrix as well. > > On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: > >> Karl: >> What matrix format do you use? Run it in parallel or sequential? >> Hong >> >> We used /proc/self/stat to track the resident set size during program >>> run, and we saw the resident set size jumped by the size of the matrix >>> right after we did matmulttranspose. >>> >>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Kun: >>>> How do you know 'MatMultTranpose creates an extra memory copy of >>>> matrix'? >>>> Hong >>>> >>>> Hi, >>>>> >>>>> >>>>> >>>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>>> >>>>> >>>>> >>>>> However we found out, MatMultTranpose create an extra memory copy of >>>>> matrix for its operation. This extra memory copy is not stated everywhere >>>>> in petsc manual. >>>>> >>>>> >>>>> >>>>> This basically double my memory requirement to solve my system. >>>>> >>>>> >>>>> >>>>> I remember mkl?s routine can do inplace matrix transpose vector >>>>> product, without transposing the matrix itself. >>>>> >>>>> >>>>> >>>>> Is this always the case? Or there is way to make petsc to do inplace >>>>> matrix transpose vector product. >>>>> >>>>> >>>>> >>>>> Any help is greatly appreciated. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Kun >>>>> >>>>> >>>>> >>>>> Schlumberger-Private >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Thu Jul 18 02:01:56 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Thu, 18 Jul 2019 16:01:56 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: Hello everyone, Since I already established a baseline to compare the cost of inserting values in PetscMatrix. And based on the hint of the number of values inserted in the matrix each time > 2) Can you tell me how many values are inserted? I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted So I made a copy of the function "MatSetValues_MPIAIJ" in " src/mat/impls/aij/mpi/mpiaij.c " and named it "MatSetValues2_MPIAIJ" I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be. I tried different number of rows to be inserted i.e.* nrow_buffer = [2, 5, 10, 20, 50, 100]* So now instead of calling "MatSetValues" for every row in the matrix , I call *"MatSetValues2_MPIAIJ"* every (nrow_buffer)th which should allow for some performance improvement the results are as follows First Remember that before 1-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 > 0 > 2-computation and insertion into eigen matrix > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 Now nrow_buffer = 2 > FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0 nrow_buffer = 5 > FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0 nrow_buffer = 10 > FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 nrow_buffer = 20 > FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 nrow_buffer = 50 > FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 nrow_buffer = 100 > FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 *As to be expected,* with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 The modifications I made based on *MatSetValues_MPIAIJ* are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage. For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough. Thanks, Kamra On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa wrote: > Regarding the first point >> >> 1) Are you timing only the insertion of values, or computation and >> insertion? > > I am timing both, the computation and insertion of values but as I said I > timed three scenarios > 1-computation only and no insertion > Computation_no_insertion 100 1.0 1.6747e-01 1.2 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0 > 2-computation and insertion into petsc matrix > >> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 >> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 >> 0 >> > 3-computation and insertion into eigen matrix > >> FilEigenMat 100 1.0 >> 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 >> 0 0 0 0 > > I timed 100 times to get a reasonably accurate timings > > as for the second point > >> 2) Can you tell me how many values are inserted? > > For a total of nearly 186062 rows per process (with 6 processes in total, > the matrix global size is 1116376) > In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining > 0.35% 2 or 3 non-zeros per row > the number of off-diagonal onnz in total is 648 nnz > So I insert nearly 4 values 186062 times ~= 744248 times per mpi process > > > Thanks, > Kamra > > On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley > wrote: > >> On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa < >> mo7ammedmostafa at gmail.com> wrote: >> >>> Sorry for the confusion >>> First I fully acknowledge that setting Matrix non-zeros or copying in >>> general is not cheap and memory access pattern can play an important role. >>> So to establish a baseline to compare with, I tried setting the same >>> matrix but in an Eigen Sparse Matrix and the timings are as follows >>> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 >>> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 * 96 * 0 0 0 0 >>> 0 >>> FilEigenMat 100 1.0 >>> 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 >>> 0 0 0 0 >>> >> >> Great. This helps. Two things would help me narrow down what is happening. >> >> 1) Are you timing only the insertion of values, or computation and >> insertion? >> >> 2) Can you tell me how many values are inserted? >> >> Thanks, >> >> Matt >> >> >>> I used the same code but simply filled a different Matrix something like >>> >>> for ( i =0; i < nRows;i++) >>> { >>> // >>> ....... >>> // Some code to get j_index, coefvalues >>> // Method1 >>> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, >>> INSERT_VALUES); >>> >>> //Method2 >>> for ( int k = 0;k < nnz_per_row; k++) >>> EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; >>> >>> } >>> Please note that only one of the two methods is being used at a time. >>> Also, I separately time the code section used to < j_index, coefvalues> >>> but simpling disabling both Method1 and Method2. >>> I found the cost to be trivial in comparison to when either one of the >>> methods is used. >>> I used Eigen out of convenience since I used for some vector and tensor >>> arithmetics somewhere else in the code and it may not be the best choice. >>> Since in PetscMatrix we technically fill two matrices: diagonal and >>> off-diagonal so I expected some difference but is that normal or am I >>> missing something. ? >>> Maybe some setting or MatOption I should be using so far this what I >>> have been using >>> >>> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, >>> PETSC_DETERMINE, >>> PETSC_DETERMINE, ptr, j , v, A); >>> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); >>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); >>> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); >>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> MatSetUp(A); >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> >>> Thanks, >>> Kamra >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 18 04:11:02 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 18 Jul 2019 09:11:02 +0000 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: Message-ID: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> 1) What preconditioner are you using? If any. 2) Where/how are you getting this information about the MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize this same sort of information to plan future optimizations. 3) Are you using more than 1 MPI rank? If you use the master branch (which we highly recommend for anyone using GPUs and PETSc) the -log_view option will log communication between CPU and GPU and display it in the summary table. This is useful for seeing exactly what operations are doing vector communication between the CPU/GPU. We welcome all feedback on the GPUs since it previously has only been lightly used. Barry > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users wrote: > > Hello everyone, > > I am new to petsc gpu and have a simple question. > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and the solve still needs some work from CPU? What are these MemCpys for during the each iteration? > > Thank you. > > Best, > Xiangdong From epscodes at gmail.com Thu Jul 18 08:11:00 2019 From: epscodes at gmail.com (Xiangdong) Date: Thu, 18 Jul 2019 09:11:00 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. wrote: > > 1) What preconditioner are you using? If any. > Currently I am using none as I want to understand how gmres works on GPU. > 2) Where/how are you getting this information about the MemCpy(HtoD) > and one call MemCpy(DtoH)? We might like to utilize this same sort of > information to plan future optimizations. > > I am using nvprof and nvvp from cuda toolkit. It looks like there are one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 case. See the attached snapshots. > 3) Are you using more than 1 MPI rank? > I tried both np=1 and np=2. Attached please find snapshots from nvvp for both np=1 and np=2 cases. The figures showing gpu calls with two pure gmres iterations. Thanks. Xiangdong > If you use the master branch (which we highly recommend for anyone using > GPUs and PETSc) the -log_view option will log communication between CPU and > GPU and display it in the summary table. This is useful for seeing exactly > what operations are doing vector communication between the CPU/GPU. > > We welcome all feedback on the GPUs since it previously has only been > lightly used. > > Barry > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hello everyone, > > > > I am new to petsc gpu and have a simple question. > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are > VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov > iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does > that mean the Krylov solve is not 100% on GPU and the solve still needs > some work from CPU? What are these MemCpys for during the each iteration? > > > > Thank you. > > > > Best, > > Xiangdong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nvprof_gmres_np1.png Type: image/png Size: 139351 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nvprof_gmres_np2.png Type: image/png Size: 303733 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Jul 18 11:11:54 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 18 Jul 2019 16:11:54 +0000 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: <95654087-5ED7-44A7-9B3F-B725A904919B@mcs.anl.gov> Thanks, these look like useful tools. Is there any way to get it to tell you where (in the function calls) the communication takes place and how much data is moved? Also ideally the amount of time spent in the communications. If the amounts are small and the times are small that is a very different story then if the amounts are large (like full vectors) and the time is large. We need to tackle the big weasels before the small ones. Barry > On Jul 18, 2019, at 8:11 AM, Xiangdong wrote: > > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. wrote: > > 1) What preconditioner are you using? If any. > > Currently I am using none as I want to understand how gmres works on GPU. > > > 2) Where/how are you getting this information about the MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize this same sort of information to plan future optimizations. > > > I am using nvprof and nvvp from cuda toolkit. It looks like there are one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 case. See the attached snapshots. > > 3) Are you using more than 1 MPI rank? > > I tried both np=1 and np=2. Attached please find snapshots from nvvp for both np=1 and np=2 cases. The figures showing gpu calls with two pure gmres iterations. > > Thanks. > Xiangdong > > > If you use the master branch (which we highly recommend for anyone using GPUs and PETSc) the -log_view option will log communication between CPU and GPU and display it in the summary table. This is useful for seeing exactly what operations are doing vector communication between the CPU/GPU. > > We welcome all feedback on the GPUs since it previously has only been lightly used. > > Barry > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users wrote: > > > > Hello everyone, > > > > I am new to petsc gpu and have a simple question. > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during each krylov iteration, there are one call MemCpy(HtoD) and one call MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and the solve still needs some work from CPU? What are these MemCpys for during the each iteration? > > > > Thank you. > > > > Best, > > Xiangdong > > From rupp at iue.tuwien.ac.at Thu Jul 18 12:14:01 2019 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Thu, 18 Jul 2019 19:14:01 +0200 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: Hi, as you can see from the screenshot, the communication is merely for scalars from the dot-products and/or norms. These are needed on the host for the control flow and convergence checks and is true for any iterative solver. Best regards, Karli On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > wrote: > > > ? ?1) What preconditioner are you using? If any. > > Currently I am using none as I want to understand how gmres works on GPU. > > > ? ?2) Where/how are you getting this information about the > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize > this same sort of information to plan future optimizations. > > I am using nvprof and nvvp from cuda toolkit. It looks like there are > one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 > case. See the attached snapshots. > > ? ?3) Are you using more than 1 MPI rank? > > > I tried both np=1 and np=2. Attached please find snapshots from nvvp for > both np=1 and np=2 cases. The figures showing gpu calls with two pure > gmres iterations. > > Thanks. > Xiangdong > > > ? If you use the master branch (which we highly recommend for > anyone using GPUs and PETSc) the -log_view option will log > communication between CPU and GPU and display it in the summary > table. This is useful for seeing exactly what operations are doing > vector communication between the CPU/GPU. > > ? We welcome all feedback on the GPUs since it previously has only > been lightly used. > > ? ?Barry > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > > wrote: > > > > Hello everyone, > > > > I am new to petsc gpu and have a simple question. > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x > are VECSEQCUDA? with GMRES(or GCR) and pcnone, I found that during > each krylov iteration, there are one call MemCpy(HtoD) and one call > MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and > the solve still needs some work from CPU? What are these MemCpys for > during the each iteration? > > > > Thank you. > > > > Best, > > Xiangdong > From rtmills at anl.gov Thu Jul 18 14:35:57 2019 From: rtmills at anl.gov (Mills, Richard Tran) Date: Thu, 18 Jul 2019 19:35:57 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: Message-ID: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Hi Kun and Karl, If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform. Best regards, Richard On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. On Wed, Jul 17, 2019 at 5:26 PM Karl Lin > wrote: MatCreateMPIAIJMKL parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong > wrote: Karl: What matrix format do you use? Run it in parallel or sequential? Hong We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users > wrote: Kun: How do you know 'MatMultTranpose creates an extra memory copy of matrix'? Hong Hi, I was using MatMultTranpose and MatMult to solver a linear system. However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. This basically double my memory requirement to solve my system. I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. Any help is greatly appreciated. Regards, Kun Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Thu Jul 18 15:09:50 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 18 Jul 2019 15:09:50 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Message-ID: Hi, Richard thank you for the detailed information. This was our guess as well. Regards, Karl On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi Kun and Karl, > > If you are using the AIJMKL matrix types and have a recent version of MKL, > the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which > are described at > > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > > The inspector-executor analysis routines take the AIJ (compressed sparse > row) format data from PETSc and then create a copy in an optimized, > internal layout used by MKL. We have to keep PETSc's own, AIJ > representation around, as it is needed for several operations that MKL does > not provide. This does, unfortunately, mean that roughly double (or more, > depending on what MKL decides to do) the amount of memory is required. The > reason you see the memory usage increase right when a MatMult() or > MatMultTranspose() operation occurs is that the we default to a "lazy" > approach to calling the analysis routine (mkl_sparse_optimize()) until an > operation that uses an MKL-provided kernel is requested. (You can use an > "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() > by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > > If memory is at enough of a premium for you that you can't afford the > extra copy used by the MKL inspector-executor routines, then I suggest > using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well > optimized for many cases (and even has some hand-optimized kernels using > Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should > try both AIJ and AIJMKL, anyway, to see which is faster for your > combination of problem and computing platform. > > Best regards, > Richard > > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: > > We also found that if we use MatCreateSeqAIJ, then no more memory increase > with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the > behavior is consistent. > > On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: > >> MatCreateMPIAIJMKL >> >> parallel and sequential exhibit the same behavior. In fact, we found that >> doing matmult will increase the memory by the size of matrix as well. >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >> >>> Karl: >>> What matrix format do you use? Run it in parallel or sequential? >>> Hong >>> >>> We used /proc/self/stat to track the resident set size during program >>>> run, and we saw the resident set size jumped by the size of the matrix >>>> right after we did matmulttranspose. >>>> >>>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Kun: >>>>> How do you know 'MatMultTranpose creates an extra memory copy of >>>>> matrix'? >>>>> Hong >>>>> >>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>>>> >>>>>> >>>>>> >>>>>> However we found out, MatMultTranpose create an extra memory copy of >>>>>> matrix for its operation. This extra memory copy is not stated everywhere >>>>>> in petsc manual. >>>>>> >>>>>> >>>>>> >>>>>> This basically double my memory requirement to solve my system. >>>>>> >>>>>> >>>>>> >>>>>> I remember mkl?s routine can do inplace matrix transpose vector >>>>>> product, without transposing the matrix itself. >>>>>> >>>>>> >>>>>> >>>>>> Is this always the case? Or there is way to make petsc to do inplace >>>>>> matrix transpose vector product. >>>>>> >>>>>> >>>>>> >>>>>> Any help is greatly appreciated. >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Kun >>>>>> >>>>>> >>>>>> >>>>>> Schlumberger-Private >>>>>> >>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 18 19:39:10 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jul 2019 00:39:10 +0000 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: <2513BABA-6047-4396-AEA1-6F0AE900B54D@anl.gov> There are a lot of moving parts here so I'm going to ask a few questions to make sure I understand the situation 1) Times a) You run the code that generates the matrix entries and simply comment out the calls to MatSetValues() and it takes .16747 seconds. b) When you include the MatSetValue calls it takes 3.8820. c) When you instead insert into an eigen matrix (which is also parallel?) it takes 2.8727 secs. d) When you in your code put all the values into a CSR matrix and then call MatUpdateMPIAIJWithArrays() it takes 2.8727 secs + .20326 secs. e) When you modify MatSetValues_MPIAIJ into MatSetValues_MPIAIJ2() that allows multiple rows that do not need matching columns it takes 2.6170 secs. 1e) When you use MatSetValues_MPIAIJ2() are you directly calling MatSetValues_MPIAIJ2() or are you still calling MatSetValues() and having it call MatSetValues_MPIAIJ2()? That is did you replace the setvalues() pointer in the function table? 1*) For the cases b,c, and e are calling MatSetValues*() multiple times for the same row or exactly once for a given row? 2) You have done the initial matrix preallocation with MatCreateMPIAIJWithArrays(); and so have perfect preallocation, that is when you run with -info and grep for malloc it says 0 mallocs are performed? 3) Are you using ADD_VALUES or INSERT_VALUES? 4) In each MatSetValues() call are you setting values into all nonzero locations of that row, or only a subset of nonzero locations in that row? Are the column indices for that row you pass in monotonically increasing or not? 5) Did you ./configure with --with-debugging=0? Based on this information I think we will understand the situation better and may have suggestions on how to proceed. Thanks, and thanks for your patience as you work to get the MatSetValues() time under control Barry > On Jul 18, 2019, at 2:01 AM, Mohammed Mostafa via petsc-users wrote: > > Hello everyone, > Since I already established a baseline to compare the cost of inserting values in PetscMatrix. > And based on the hint of the number of values inserted in the matrix each time > 2) Can you tell me how many values are inserted? > I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for > FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same > > So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted > So I made a copy of the function "MatSetValues_MPIAIJ" in "src/mat/impls/aij/mpi/mpiaij.c" and named it "MatSetValues2_MPIAIJ" > I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices > > So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be. > I tried different number of rows to be inserted i.e. nrow_buffer = [2, 5, 10, 20, 50, 100] > So now instead of calling "MatSetValues" for every row in the matrix , I call "MatSetValues2_MPIAIJ" every (nrow_buffer)th which should allow for some performance improvement > the results are as follows > First Remember that before > 1-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > 2-computation and insertion into eigen matrix > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > Now > nrow_buffer = 2 > FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0 > nrow_buffer = 5 > FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0 > nrow_buffer = 10 > FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 20 > FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 50 > FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 100 > FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > As to be expected, with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 > The modifications I made based on MatSetValues_MPIAIJ are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage. > For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough. > > Thanks, > Kamra > > On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa wrote: > Regarding the first point > 1) Are you timing only the insertion of values, or computation and insertion? > I am timing both, the computation and insertion of values but as I said I timed three scenarios > 1-computation only and no insertion > Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0 > 2-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > 3-computation and insertion into eigen matrix > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > I timed 100 times to get a reasonably accurate timings > > as for the second point > 2) Can you tell me how many values are inserted? > For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376) > In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row > the number of off-diagonal onnz in total is 648 nnz > So I insert nearly 4 values 186062 times ~= 744248 times per mpi process > > > Thanks, > Kamra > > On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley wrote: > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa wrote: > Sorry for the confusion > First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role. > So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > Great. This helps. Two things would help me narrow down what is happening. > > 1) Are you timing only the insertion of values, or computation and insertion? > > 2) Can you tell me how many values are inserted? > > Thanks, > > Matt > > I used the same code but simply filled a different Matrix something like > > for ( i =0; i < nRows;i++) > { > // > ....... > // Some code to get j_index, coefvalues > // Method1 > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); > > //Method2 > for ( int k = 0;k < nnz_per_row; k++) > EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; > > } > Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2. > I found the cost to be trivial in comparison to when either one of the methods is used. > I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice. > Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ? > Maybe some setting or MatOption I should be using so far this what I have been using > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE, > PETSC_DETERMINE, ptr, j , v, A); > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > MatSetUp(A); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > Thanks, > Kamra > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From bsmith at mcs.anl.gov Thu Jul 18 19:48:12 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jul 2019 00:48:12 +0000 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: Message-ID: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> 1) Could you please send MatSetValues2_MPIAIJ()? 2) What time do you get if you use MatSetValues2_MPIAIJ() with an nrow_buffer = 1 ? Thanks Barry > On Jul 18, 2019, at 2:01 AM, Mohammed Mostafa via petsc-users wrote: > > Hello everyone, > Since I already established a baseline to compare the cost of inserting values in PetscMatrix. > And based on the hint of the number of values inserted in the matrix each time > 2) Can you tell me how many values are inserted? > I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for > FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same > > So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted > So I made a copy of the function "MatSetValues_MPIAIJ" in "src/mat/impls/aij/mpi/mpiaij.c" and named it "MatSetValues2_MPIAIJ" > I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices > > So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be. > I tried different number of rows to be inserted i.e. nrow_buffer = [2, 5, 10, 20, 50, 100] > So now instead of calling "MatSetValues" for every row in the matrix , I call "MatSetValues2_MPIAIJ" every (nrow_buffer)th which should allow for some performance improvement > the results are as follows > First Remember that before > 1-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > 2-computation and insertion into eigen matrix > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > Now > nrow_buffer = 2 > FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0 > nrow_buffer = 5 > FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0 > nrow_buffer = 10 > FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 20 > FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 50 > FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > nrow_buffer = 100 > FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > As to be expected, with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 > The modifications I made based on MatSetValues_MPIAIJ are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage. > For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough. > > Thanks, > Kamra > > On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa wrote: > Regarding the first point > 1) Are you timing only the insertion of values, or computation and insertion? > I am timing both, the computation and insertion of values but as I said I timed three scenarios > 1-computation only and no insertion > Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0 > 2-computation and insertion into petsc matrix > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > 3-computation and insertion into eigen matrix > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > I timed 100 times to get a reasonably accurate timings > > as for the second point > 2) Can you tell me how many values are inserted? > For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376) > In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row > the number of off-diagonal onnz in total is 648 nnz > So I insert nearly 4 values 186062 times ~= 744248 times per mpi process > > > Thanks, > Kamra > > On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley wrote: > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa wrote: > Sorry for the confusion > First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role. > So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > Great. This helps. Two things would help me narrow down what is happening. > > 1) Are you timing only the insertion of values, or computation and insertion? > > 2) Can you tell me how many values are inserted? > > Thanks, > > Matt > > I used the same code but simply filled a different Matrix something like > > for ( i =0; i < nRows;i++) > { > // > ....... > // Some code to get j_index, coefvalues > // Method1 > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); > > //Method2 > for ( int k = 0;k < nnz_per_row; k++) > EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; > > } > Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2. > I found the cost to be trivial in comparison to when either one of the methods is used. > I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice. > Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ? > Maybe some setting or MatOption I should be using so far this what I have been using > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE, > PETSC_DETERMINE, ptr, j , v, A); > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > MatSetUp(A); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > Thanks, > Kamra > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From epscodes at gmail.com Thu Jul 18 20:45:11 2019 From: epscodes at gmail.com (Xiangdong) Date: Thu, 18 Jul 2019 21:45:11 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: Yes, nvprof can give the size of the data as well as the amount of time for data movement. See the attached snapshots. I can understand some of the numbers, but not the HtoD case. In DtoH1, it is the data movement from VecMDot. The size of data is 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = 8*128*8 = 8192. My question is: instead of calling cublasDdot nv times, why do you implement your own kernels? I guess it must be for performance, but can you explain a little more? In DtoH2, it is the data movement from VecNorm. The size of data is 8B, which is just the sizeof(PetscScalar). In DtoD1, it is the data movement from VecAXPY. The size of data is 17.952MB, which is exactly sizeof(PetscScalar)*length(b). However, I do not understand the number in HostToDevice in gmres for np=1. The size of data movement is 1.032KB. I thought this is related to the updated upper Hessenberg matrix, but the number does not match. Can anyone help me understand the data movement of HToD in GMRES for np=1? Thank you. Best, Xiangdong On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp wrote: > Hi, > > as you can see from the screenshot, the communication is merely for > scalars from the dot-products and/or norms. These are needed on the host > for the control flow and convergence checks and is true for any > iterative solver. > > Best regards, > Karli > > > > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > > > > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > > wrote: > > > > > > 1) What preconditioner are you using? If any. > > > > Currently I am using none as I want to understand how gmres works on GPU. > > > > > > 2) Where/how are you getting this information about the > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize > > this same sort of information to plan future optimizations. > > > > I am using nvprof and nvvp from cuda toolkit. It looks like there are > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 > > case. See the attached snapshots. > > > > 3) Are you using more than 1 MPI rank? > > > > > > I tried both np=1 and np=2. Attached please find snapshots from nvvp for > > both np=1 and np=2 cases. The figures showing gpu calls with two pure > > gmres iterations. > > > > Thanks. > > Xiangdong > > > > > > If you use the master branch (which we highly recommend for > > anyone using GPUs and PETSc) the -log_view option will log > > communication between CPU and GPU and display it in the summary > > table. This is useful for seeing exactly what operations are doing > > vector communication between the CPU/GPU. > > > > We welcome all feedback on the GPUs since it previously has only > > been lightly used. > > > > Barry > > > > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > > > wrote: > > > > > > Hello everyone, > > > > > > I am new to petsc gpu and have a simple question. > > > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during > > each krylov iteration, there are one call MemCpy(HtoD) and one call > > MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and > > the solve still needs some work from CPU? What are these MemCpys for > > during the each iteration? > > > > > > Thank you. > > > > > > Best, > > > Xiangdong > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DtoH1.png Type: image/png Size: 24407 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DtoH2.png Type: image/png Size: 22221 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DtoD1.png Type: image/png Size: 24194 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: HtoD1.png Type: image/png Size: 23920 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Jul 18 22:24:00 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jul 2019 03:24:00 +0000 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: <12B91276-38F1-469B-A98B-3905AFE5C9AB@mcs.anl.gov> Thanks for your email. > On Jul 18, 2019, at 8:45 PM, Xiangdong wrote: > > Yes, nvprof can give the size of the data as well as the amount of time for data movement. See the attached snapshots. > > I can understand some of the numbers, but not the HtoD case. > > In DtoH1, it is the data movement from VecMDot. The size of data is 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = 8*128*8 = 8192. My question is: instead of calling cublasDdot nv times, why do you implement your own kernels? I guess it must be for performance, but can you explain a little more? It is two fold: reduce the number of kernel launches and reduce the number of times the x vector needs to be streamed from memory to floating point units. In some cases our code produces much faster times but it has not been studied extensively and there may be better ways. https://bitbucket.org/petsc/petsc/issues/319/revisit-optimizations-of-vecmdot_cuda > > In DtoH2, it is the data movement from VecNorm. The size of data is 8B, which is just the sizeof(PetscScalar). > > In DtoD1, it is the data movement from VecAXPY. The size of data is 17.952MB, which is exactly sizeof(PetscScalar)*length(b). > > However, I do not understand the number in HostToDevice in gmres for np=1. The size of data movement is 1.032KB. I thought this is related to the updated upper Hessenberg matrix, but the number does not match. Can anyone help me understand the data movement of HToD in GMRES for np=1? You'll need to look at the GMRES code to get the numbers exactly right. The results of the MDot are stored on the CPU and then copied back down to the GPU for the VecMAXPY(). In addition KSPGMRESBuildSoln() has possibly another VecMAXPY() plus a some vector operations for example ierr = VecAXPY(vdest,1.0,VEC_TEMP);CHKERRQ(ierr); note that even though the 1.0 is a constant at compile time the VecAXPY_SeqCUDA() doesn't know this and has to bring the alpha value down from the CPU and this requires a memory communication of 8 bytes. How many iterations did you run GMRES? Based on the MDot communication of 8192 bytes/128 (MDOT_WORKGROUP_SIZE) = 64 bytes it looks like 8 scalars are computed in mdots? But the HtoD is 1032 bytes which divided by 8 bytes is 129 scalars communicated. I cannot explain why the number going down seems much more than going up. Perhaps you can have your tool zoom in more to see in exactly which routines the values are going down in? You'll need to dig around a bit deeper to attach each down communication to its place in the code. The Hessenberg matrix is never moved between the GPU and CPU, it just stays in CPU. The assumption is that it is so small doing its computation on the GPU would not be faster. > > Thank you. > > Best, > Xiangdong > > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp wrote: > Hi, > > as you can see from the screenshot, the communication is merely for > scalars from the dot-products and/or norms. These are needed on the host > for the control flow and convergence checks and is true for any > iterative solver. > > Best regards, > Karli > > > > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > > > > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > > wrote: > > > > > > 1) What preconditioner are you using? If any. > > > > Currently I am using none as I want to understand how gmres works on GPU. > > > > > > 2) Where/how are you getting this information about the > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize > > this same sort of information to plan future optimizations. > > > > I am using nvprof and nvvp from cuda toolkit. It looks like there are > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 > > case. See the attached snapshots. > > > > 3) Are you using more than 1 MPI rank? > > > > > > I tried both np=1 and np=2. Attached please find snapshots from nvvp for > > both np=1 and np=2 cases. The figures showing gpu calls with two pure > > gmres iterations. > > > > Thanks. > > Xiangdong > > > > > > If you use the master branch (which we highly recommend for > > anyone using GPUs and PETSc) the -log_view option will log > > communication between CPU and GPU and display it in the summary > > table. This is useful for seeing exactly what operations are doing > > vector communication between the CPU/GPU. > > > > We welcome all feedback on the GPUs since it previously has only > > been lightly used. > > > > Barry > > > > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > > > wrote: > > > > > > Hello everyone, > > > > > > I am new to petsc gpu and have a simple question. > > > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b and x > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that during > > each krylov iteration, there are one call MemCpy(HtoD) and one call > > MemCpy(DtoH). Does that mean the Krylov solve is not 100% on GPU and > > the solve still needs some work from CPU? What are these MemCpys for > > during the each iteration? > > > > > > Thank you. > > > > > > Best, > > > Xiangdong > > > From mo7ammedmostafa at gmail.com Fri Jul 19 01:34:30 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Fri, 19 Jul 2019 15:34:30 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> References: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> Message-ID: > > 1) Times > a) You run the code that generates the matrix entries and simply > comment out the calls to MatSetValues() and it takes > .16747 seconds. Yes > > b) When you include the MatSetValue calls it takes 3.8820. > Yes > c) When you instead insert into an eigen matrix (which is also > parallel?) > No, The Eigen Matrix is just a local matrix to each process and it doesn't support MPI ( at least how I use it ), I just used to establish a baseline for the insertion cost in CSR matrix ( since I don't set off-processor rows anyway) And it both cases it is a local insertion until MatAssembly where communication and inter-processor copy should happen After I insert in Eigen and I copy again the Eigen matrix to the Petsc Matrix, using *MatUpdateMPIAIJWithArrays() which costs me an additional 0*.20326 secs.*,* which is what I intend to use for linear system solution using ksp, pc > it takes 2.8727 secs. > d) When you in your code put all the values into a CSR matrix and > then call MatUpdateMPIAIJWithArrays() it takes > 2.8727 secs + .20326 secs. > Yes > e) When you modify MatSetValues_MPIAIJ into MatSetValues_MPIAIJ2() > that allows multiple rows that do not need matching > columns it takes 2.6170 secs. Yes 1e) When you use MatSetValues_MPIAIJ2() are you directly calling MatSetValues_MPIAIJ2() or are you still calling MatSetValues() and having it call MatSetValues_MPIAIJ2()? That is did you replace the setvalues() pointer in the function table? Yes, I am calling *MatSetValues2_MPIAIJ()* directly not *MatSetValues()*, I just wanted to confirm the behavior first and if I am getting any performance gain I wanted to be as minimally intrusive to the code as possible right now to avoid causing problems elsewhere since I have just started learning how it works. 1*) For the cases b,c, and e are calling MatSetValues*() multiple times for > the same row or exactly once for a given row? I am only calling MatSetValues exactly once per row and each time I set 4 nonzero values although for the future I may set as many as 7 non-zero values however, in *MatSetValues2_MPIAIJ() I call it once every *nrow_buffer The call is like this PetscLogEventBegin(FillPetscMatEvent, 0, 0, 0, 0); auto mat = &PhiEqnSolver.mat; int buffer_size = 100, n_inner, Rows_left,nj_per_row_i; int n_outer = int(ceil(double(nCells) / double(buffer_size))); cell = 0; Rows_left = nCells; for (int i_outer = 0; i_outer < n_outer; i_outer++) { n_inner = min(buffer_size,Rows_left ); k = 0; for (i = 0; i < n_inner; i++) { cell_gindex = cellGIndex[cell]; i_index[i] = cell_gindex; // set the system RHS = f * vol Eqn_rhs_ptr[cell] += cellVol[cell] * rhsFieldptr[cell]; istart = cellIFace[cell]; iend = cellIFace[cell + 1]; diagCOEF = 0.0; nj_per_row_i =0; for (fptr = istart; fptr < iend; fptr++) { face = cellFace[fptr]; nghbr = faceNeighbor[face]; COEF = Coef[face]; // flux coef computed on the faces of each cell in the domain if (nghbr < 0) { bc_type = faceBCType[face]; if (bc_type == PROCESSOR) { neighbor_cell = facePhysTag[face]; j_index[k] = neighbor_cell; coefvalues[k] = COEF; diagCOEF -= COEF; k++; nj_per_row_i++; } else { diagCOEF -= COEF; } } else { owner = faceOwner[face]; neighbor_cell = owner + nghbr - cell; j_index[k] = cellGIndex[neighbor_cell]; coefvalues[k] = COEF; diagCOEF -= COEF; k++; nj_per_row_i++; } } j_index[k] = cell_gindex; coefvalues[k] = diagCOEF; k++; nj_per_row_i++; nj_perRow[i] = nj_per_row_i; cell++; Rows_left--; } MatSetValues2_MPIAIJ(*A, n_inner, i_index, nj_perRow, j_index, coefvalues, INSERT_VALUES); } PetscLogEventEnd(FillPetscMatEvent, 0, 0, 0, 0); 2) You have done the initial matrix preallocation with > MatCreateMPIAIJWithArrays(); and so have perfect preallocation, that is > when you run with > -info and grep for malloc it says 0 mallocs are performed? Yes no mallocs and no unused non-zeros check the attached file 3) Are you using ADD_VALUES or INSERT_VALUES? here are the times for nrow_buffer = 50 With InsertValues FillCSTMat_with_MatSetValues 100 1.0 2.5512e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 WithAddValues FillPetscMat 100 1.0 2.6654e+00 1.0 7.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00 17 53 0 0 0 90100 0 0 0 167 Additionally I have to zero the matrix which costs MatZeroEntries 100 1.0 9.5259e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 3 0 0 0 0 0 INSERT_VALUES is abit faster than ADD_VALUES the difference I saw can be attributed to the cost computation ( addition) > 4) In each MatSetValues() call are you setting values into all nonzero > locations of that row, or only a subset of nonzero locations in that row? > Are the column indices for that row you pass in monotonically > increasing or not? Yes I setting new values for all non-zero values in that row, not a subset of it No, the column indices are Not monotonically increasing as you can see from the above code the column indices come from the mesh connectivity come. I suppose I can sort them and make line an ordering map for each row and store it and use that to rearrange the values before inserting them. But would it really matter that much ?! From memory access point, I suppose it could matter. 5) Did you ./configure with --with-debugging=0? PETSC_ARCH=release3 *-with-debugging=0* COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre As for the second Message > 1) Could you please send nrow_buffer = 1 > Yes, Of course. As I said I made very small changes and now I trying to dig deeper into the code to see if there is more to do Should I attach the modified < mpiaij.c and petscmat.h > to this message thread or send it directly to you > 2) What time do you get if you use MatSetValues2_MPIAIJ() with an > nrow_buffer = 1 ? Honestly, I didn't try that before. First I replaced the call in the code above to* MatSetValues() *and set nrow_buffer = 1 so it should be equivalent to before and here is the time FillPetscMat 100 1.0 *3.6594e+00* 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 22 0 0 0 0 96 0 0 0 0 0 Now I used *MatSetValues2_MPIAIJ()* with an nrow_buffer = 1 FillPetscMat 100 1.0* 3.3920e+00* 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 96 0 0 0 0 0 For some reason, *MatSetValues2_MPIAIJ *is a bit faster. Sorry for the long reply. Thanks, Kamra On Fri, Jul 19, 2019 at 9:48 AM Smith, Barry F. wrote: > > 1) Could you please send MatSetValues2_MPIAIJ()? > > 2) What time do you get if you use MatSetValues2_MPIAIJ() with an > nrow_buffer = 1 ? > > Thanks > > Barry > > > > On Jul 18, 2019, at 2:01 AM, Mohammed Mostafa via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hello everyone, > > Since I already established a baseline to compare the cost of inserting > values in PetscMatrix. > > And based on the hint of the number of values inserted in the matrix > each time > > 2) Can you tell me how many values are inserted? > > I took a look at the source code for "MatSetValues_MPIAIJ" and found > that it seems to be designed for > > FEM assembly of the global matrix from element matrices(from what I > remember from undergrad) since it requires for the setting of multiple rows > that the col indices should be the same > > > > So to increase the number of inserted values I need to modify the > implementation of MatSetValues_MPIAIJ to allow for more values to be > inserted > > So I made a copy of the function "MatSetValues_MPIAIJ" in > "src/mat/impls/aij/mpi/mpiaij.c" and named it "MatSetValues2_MPIAIJ" > > I made some minor changes to allow for inserting multiple rows > regardless of whether they have the same col indices > > > > So what I do now is store the data for multiple rows and then insert > them all together and I figured I would see how the performance would be. > > I tried different number of rows to be inserted i.e. nrow_buffer = [2, > 5, 10, 20, 50, 100] > > So now instead of calling "MatSetValues" for every row in the matrix , > I call "MatSetValues2_MPIAIJ" every (nrow_buffer)th which should allow for > some performance improvement > > the results are as follows > > First Remember that before > > 1-computation and insertion into petsc matrix > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > 2-computation and insertion into eigen matrix > > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 > > > > Now > > nrow_buffer = 2 > > FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0 > > nrow_buffer = 5 > > FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0 > > nrow_buffer = 10 > > FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 20 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 50 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 100 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > > > As to be expected, with increasing the number of rows to be inserted the > overhead reduces until it basically stagnates somewhere between 20-50 > > The modifications I made based on MatSetValues_MPIAIJ are very small but > the effect is significant ( drop in insertion cost by 33%) and it is now > even faster than Eigen(baseline) at insertion with my naive usage. > > For now I am quite satisfied with the outcome. There is probably some > room for improvement but for now this is enough. > > > > Thanks, > > Kamra > > > > On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > > Regarding the first point > > 1) Are you timing only the insertion of values, or computation and > insertion? > > I am timing both, the computation and insertion of values but as I said > I timed three scenarios > > 1-computation only and no insertion > > Computation_no_insertion 100 1.0 1.6747e-01 > 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 > 0 > > 2-computation and insertion into petsc matrix > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > 3-computation and insertion into eigen matrix > > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 > > I timed 100 times to get a reasonably accurate timings > > > > as for the second point > > 2) Can you tell me how many values are inserted? > > For a total of nearly 186062 rows per process (with 6 processes in > total, the matrix global size is 1116376) > > In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining > 0.35% 2 or 3 non-zeros per row > > the number of off-diagonal onnz in total is 648 nnz > > So I insert nearly 4 values 186062 times ~= 744248 times per mpi process > > > > > > Thanks, > > Kamra > > > > On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley > wrote: > > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa < > mo7ammedmostafa at gmail.com> wrote: > > Sorry for the confusion > > First I fully acknowledge that setting Matrix non-zeros or copying in > general is not cheap and memory access pattern can play an important role. > > So to establish a baseline to compare with, I tried setting the same > matrix but in an Eigen Sparse Matrix and the timings are as follows > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > FilEigenMat 100 1.0 > 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 > 0 0 0 0 > > > > Great. This helps. Two things would help me narrow down what is > happening. > > > > 1) Are you timing only the insertion of values, or computation and > insertion? > > > > 2) Can you tell me how many values are inserted? > > > > Thanks, > > > > Matt > > > > I used the same code but simply filled a different Matrix something like > > > > for ( i =0; i < nRows;i++) > > { > > // > > ....... > > // Some code to get j_index, coefvalues > > // Method1 > > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, > INSERT_VALUES); > > > > //Method2 > > for ( int k = 0;k < nnz_per_row; k++) > > EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; > > > > } > > Please note that only one of the two methods is being used at a time. > Also, I separately time the code section used to < j_index, coefvalues> > but simpling disabling both Method1 and Method2. > > I found the cost to be trivial in comparison to when either one of the > methods is used. > > I used Eigen out of convenience since I used for some vector and tensor > arithmetics somewhere else in the code and it may not be the best choice. > > Since in PetscMatrix we technically fill two matrices: diagonal and > off-diagonal so I expected some difference but is that normal or am I > missing something. ? > > Maybe some setting or MatOption I should be using so far this what I > have been using > > > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, > PETSC_DETERMINE, > > PETSC_DETERMINE, ptr, j , v, A); > > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > > MatSetUp(A); > > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > > Thanks, > > Kamra > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [5] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [1] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [2] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [3] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [4] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 0 Total message length 0 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 930313; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 930312; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [2] MatSeqAIJCheckInode(): Found 38030 nodes of 186064. Limit used: 5. Using Inode routines [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 930313; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines. [4] MatSeqAIJCheckInode(): Found 37991 nodes of 186063. Limit used: 5. Using Inode routines [3] MatSeqAIJCheckInode(): Found 37997 nodes of 186063. Limit used: 5. Using Inode routines [0] MatSeqAIJCheckInode(): Found 37993 nodes of 186062. Limit used: 5. Using Inode routines [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [1] MatSeqAIJCheckInode(): Found 38413 nodes of 186062. Limit used: 5. Using Inode routines [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 930314; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines. [5] MatSeqAIJCheckInode(): Found 38565 nodes of 186062. Limit used: 5. Using Inode routines [3] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930328 [4] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930328 [2] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930327 [1] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [0] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [5] MatCreateSubMatrices_MPIAIJ_Local(): Number of outgoing messages 5 Total message length 930329 [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,1137 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 929177)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [5] MatSeqAIJCheckInode(): Found 187410 nodes of 930314. Limit used: 5. Using Inode routines [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 930312 X 186064; storage space: 0 unneeded,685 used [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 929627)/(num_localrows 930312) > 0.6. Use CompressedRow routines. [2] MatSeqAIJCheckInode(): Found 186870 nodes of 930312. Limit used: 5. Using Inode routines [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,1011 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 929303)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [1] MatSeqAIJCheckInode(): Found 187283 nodes of 930314. Limit used: 5. Using Inode routines [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 930313 X 186063; storage space: 0 unneeded,658 used [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 929655)/(num_localrows 930313) > 0.6. Use CompressedRow routines. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 930313 X 186063; storage space: 0 unneeded,649 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 929664)/(num_localrows 930313) > 0.6. Use CompressedRow routines. [3] MatSeqAIJCheckInode(): Found 186842 nodes of 930313. Limit used: 5. Using Inode routines [4] MatSeqAIJCheckInode(): Found 186825 nodes of 930313. Limit used: 5. Using Inode routines [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 930314 X 186062; storage space: 0 unneeded,648 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 929666)/(num_localrows 930314) > 0.6. Use CompressedRow routines. [0] MatSeqAIJCheckInode(): Found 186837 nodes of 930314. Limit used: 5. Using Inode routines From eda.oktay at metu.edu.tr Fri Jul 19 01:57:52 2019 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 19 Jul 2019 09:57:52 +0300 Subject: [petsc-users] Imbalance in ParMETIS In-Reply-To: References: Message-ID: Dear Matt, You were right, I confused concepts. By setting vertex weights, I am able to get different number of edge cuts. Thanks, Eda Matthew Knepley , 1 Tem 2019 Pzt, 13:10 tarihinde ?unu yazd?: > On Mon, Jul 1, 2019 at 3:00 AM Eda Oktay wrote: > >> Dear Matt, >> >> In order to get a nice cut, I am trying to change the balance, that is >> why I tried to change ubvec. But still I couldn't understand why it didn't >> change anything. >> >> I tried to change vertex weights but it only made Max and Min change but >> it didn't change the cut. >> > > I think you are confusing concepts. The "cut" is the number of edges > discarded when you divide the mesh > into pieces, but each piece is supposed to have an equal size. The only > way you can get the partitioner to > divide the mesh into pieces of unequal size is to use vertex weights. > > Thanks, > > Matt > > >> Thanks, >> >> Eda >> >> >> On Fri, Jun 28, 2019, 1:49 PM Matthew Knepley wrote: >> >>> On Fri, Jun 28, 2019 at 5:38 AM Eda Oktay via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hello everyone, >>>> >>>> I am trying to change load balance in ParMETIS in order to give >>>> different number of elements to different processors. In the manual, it >>>> states that changing "ubvec" parameter should work. So, instead of 1.05, I >>>> gave 1.2,1.5 and so on but load balance didn't change. I always get Max 5 >>>> Min 5 when I view load balance. The partitioning part of my code is: >>>> >>> >>> ParMetis is always balancing. ubvec changes the balance between having >>> even partitions and a nice cut. If >>> you want unbalanced partitions, you have to change the vertex weights. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> ierr = MatConvert(A,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr); >>>> >>>> ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr); >>>> ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr); >>>> >>>> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr); >>>> ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr); >>>> MatPartitioningViewImbalance(part,partitioning); >>>> >>>> What should I do? Why changing ubvec didn't change anything, am I >>>> missing something? >>>> >>>> Thanks, >>>> >>>> Eda >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Fri Jul 19 02:01:25 2019 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Fri, 19 Jul 2019 09:01:25 +0200 Subject: [petsc-users] How to stop a TS early Message-ID: Dear Petsc users, I am facing the following situation. I run a code that uses a Petsc TimeStepper on a supercomputer that has a given walltime. In real time, I am able to get the remaining time, and when I have only a few minutes left I want to stop the code in a clean fashion, rather than an abrupt out-of-time. I set-up all the machinery to check the remaining time already in the TSPostStep so the code checks after each iteration, and sets a flag to 1 when it got to stop. Now what I would like to know, is how to tell TSSolve to stop iterating when that flag is set to 1 in the TSPostStep ? Any clues would be much appreciated ! Best regards, Thibault Bridel-Bertomeu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Jul 19 10:33:31 2019 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Jul 2019 18:33:31 +0300 Subject: [petsc-users] How to stop a TS early In-Reply-To: References: Message-ID: PetscErrorCode YourPostStepCallback(...) { ... ierr = TSSetConvergedReason(ts,TS_CONVERGED_USER);CHKERRQ(ierr); ... } On Fri, 19 Jul 2019 at 18:21, Thibault Bridel-Bertomeu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear Petsc users, > > I am facing the following situation. > I run a code that uses a Petsc TimeStepper on a supercomputer that has a > given walltime. In real time, I am able to get the remaining time, and when > I have only a few minutes left I want to stop the code in a clean fashion, > rather than an abrupt out-of-time. I set-up all the machinery to check the > remaining time already in the TSPostStep so the code checks after each > iteration, and sets a flag to 1 when it got to stop. > > Now what I would like to know, is how to tell TSSolve to stop iterating > when that flag is set to 1 in the TSPostStep ? > > Any clues would be much appreciated ! > > Best regards, > > Thibault Bridel-Bertomeu > -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 19 10:52:01 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jul 2019 10:52:01 -0500 Subject: [petsc-users] Imbalance in ParMETIS In-Reply-To: References: Message-ID: On Fri, Jul 19, 2019 at 1:58 AM Eda Oktay wrote: > Dear Matt, > > You were right, I confused concepts. By setting vertex weights, I am able > to get different number of edge cuts. > Great! I am glad it works now. Thanks, Matt > Thanks, > > Eda > > Matthew Knepley , 1 Tem 2019 Pzt, 13:10 tarihinde ?unu > yazd?: > >> On Mon, Jul 1, 2019 at 3:00 AM Eda Oktay wrote: >> >>> Dear Matt, >>> >>> In order to get a nice cut, I am trying to change the balance, that is >>> why I tried to change ubvec. But still I couldn't understand why it didn't >>> change anything. >>> >>> I tried to change vertex weights but it only made Max and Min change but >>> it didn't change the cut. >>> >> >> I think you are confusing concepts. The "cut" is the number of edges >> discarded when you divide the mesh >> into pieces, but each piece is supposed to have an equal size. The only >> way you can get the partitioner to >> divide the mesh into pieces of unequal size is to use vertex weights. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Eda >>> >>> >>> On Fri, Jun 28, 2019, 1:49 PM Matthew Knepley wrote: >>> >>>> On Fri, Jun 28, 2019 at 5:38 AM Eda Oktay via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I am trying to change load balance in ParMETIS in order to give >>>>> different number of elements to different processors. In the manual, it >>>>> states that changing "ubvec" parameter should work. So, instead of 1.05, I >>>>> gave 1.2,1.5 and so on but load balance didn't change. I always get Max 5 >>>>> Min 5 when I view load balance. The partitioning part of my code is: >>>>> >>>> >>>> ParMetis is always balancing. ubvec changes the balance between having >>>> even partitions and a nice cut. If >>>> you want unbalanced partitions, you have to change the vertex weights. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> ierr = MatConvert(A,MATMPIADJ,MAT_INITIAL_MATRIX,&AL);CHKERRQ(ierr); >>>>> >>>>> ierr = MatPartitioningCreate(MPI_COMM_WORLD,&part);CHKERRQ(ierr); >>>>> ierr = MatPartitioningSetAdjacency(part,AL);CHKERRQ(ierr); >>>>> >>>>> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr); >>>>> ierr = MatPartitioningApply(part,&partitioning);CHKERRQ(ierr); >>>>> MatPartitioningViewImbalance(part,partitioning); >>>>> >>>>> What should I do? Why changing ubvec didn't change anything, am I >>>>> missing something? >>>>> >>>>> Thanks, >>>>> >>>>> Eda >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Fri Jul 19 11:08:04 2019 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Fri, 19 Jul 2019 18:08:04 +0200 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: Hi Xiangdong, > I can understand some of the numbers, but not the HtoD case. > > In DtoH1, it is the data movement from VecMDot. The size of data is > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = 8*128*8 > = 8192. My question is: instead of calling cublasDdot nv times, why do > you implement your own kernels? I guess it must be for performance, but > can you explain a little more? Yes, this is a performance optimization. We've used several dot-products (suffers from kernel launch latency) as well as matrix-vector-products (suffers extra matrix setup) in the past; in both cases, there was extra memory traffic, thus impacting performance. The reason why the data size is 8192 is to get around a separate reduction stage on the GPU (i.e. a second kernel launch). By moving the data to the CPU and doing the reduction there, one is faster than doing it on the GPU and then moving only a few numbers. This has to do with PCI-Express latency: It takes about the same time to send a single byte as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes the limiting factor. > In DtoH2, it is the data movement from VecNorm. The size of data is 8B, > which is just the sizeof(PetscScalar). This is most likely the result required for the control flow on the CPU. > In DtoD1, it is the data movement from VecAXPY. The size of data is > 17.952MB, which is exactly sizeof(PetscScalar)*length(b). This is a vector assignment. If I remember correctly, it uses the memcpy-routines and hence shows up as a separate DtoD instead of just a kernel. It matches the time required for scal_kernel_val (scaling a vector by a scalar), so it runs at full bandwidth on the GPU. > However, I do not understand the number in HostToDevice in gmres for > np=1. The size of data movement is 1.032KB. I thought this is related to > the updated upper Hessenberg matrix, but the number does not match. Can > anyone help me understand the data movement of HToD in GMRES for np=1? 1032 = (128+1)*8, so this might be some auxiliary work information on the GPU. I could figure out the exact source of these transfers, but that is some effort. Let me know whether this is important information for you, then I can do it. Best regards, Karli > > Thank you. > > Best, > Xiangdong > > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp > wrote: > > Hi, > > as you can see from the screenshot, the communication is merely for > scalars from the dot-products and/or norms. These are needed on the > host > for the control flow and convergence checks and is true for any > iterative solver. > > Best regards, > Karli > > > > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > > > > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > > > >> wrote: > > > > > >? ? ? ? ?1) What preconditioner are you using? If any. > > > > Currently I am using none as I want to understand how gmres works > on GPU. > > > > > >? ? ? ? ?2) Where/how are you getting this information about the > >? ? ?MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to utilize > >? ? ?this same sort of information to plan future optimizations. > > > > I am using nvprof and nvvp from cuda toolkit. It looks like there > are > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for np=1 > > case. See the attached snapshots. > > > >? ? ? ? ?3) Are you using more than 1 MPI rank? > > > > > > I tried both np=1 and np=2. Attached please find snapshots from > nvvp for > > both np=1 and np=2 cases. The figures showing gpu calls with two > pure > > gmres iterations. > > > > Thanks. > > Xiangdong > > > > > >? ? ? ? If you use the master branch (which we highly recommend for > >? ? ?anyone using GPUs and PETSc) the -log_view option will log > >? ? ?communication between CPU and GPU and display it in the summary > >? ? ?table. This is useful for seeing exactly what operations are > doing > >? ? ?vector communication between the CPU/GPU. > > > >? ? ? ? We welcome all feedback on the GPUs since it previously > has only > >? ? ?been lightly used. > > > >? ? ? ? ?Barry > > > > > >? ? ? > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > >? ? ? > >> > wrote: > >? ? ? > > >? ? ? > Hello everyone, > >? ? ? > > >? ? ? > I am new to petsc gpu and have a simple question. > >? ? ? > > >? ? ? > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b > and x > >? ? ?are VECSEQCUDA? with GMRES(or GCR) and pcnone, I found that > during > >? ? ?each krylov iteration, there are one call MemCpy(HtoD) and > one call > >? ? ?MemCpy(DtoH). Does that mean the Krylov solve is not 100% on > GPU and > >? ? ?the solve still needs some work from CPU? What are these > MemCpys for > >? ? ?during the each iteration? > >? ? ? > > >? ? ? > Thank you. > >? ? ? > > >? ? ? > Best, > >? ? ? > Xiangdong > > > From michael.wick.1980 at gmail.com Fri Jul 19 11:10:11 2019 From: michael.wick.1980 at gmail.com (Michael Wick) Date: Fri, 19 Jul 2019 09:10:11 -0700 Subject: [petsc-users] confused by the converged reason output Message-ID: Hi PETSc team: I am a bit confused by the output of converged reason. I set the relative tolerance to be 10^{-3}. In my run, I get the monitor true residual to be as follows. 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 And the converged reason is solve converged due to CONVERGED_RTOL iterations 3. The relative error is quite far from the prescribed tolerance. Is there something I should know about the stopping criteria? The ks_view output is Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 KSP Object:(m_fieldsplit_0_) 24 MPI processes type: gmres GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=100, initial guess is zero tolerances: relative=0.001, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object:(m_fieldsplit_0_) 24 MPI processes type: hypre HYPRE BoomerAMG preconditioning HYPRE BoomerAMG: Cycle type V HYPRE BoomerAMG: Maximum number of levels 25 HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 HYPRE BoomerAMG: Threshold for strong coupling 0.25 HYPRE BoomerAMG: Interpolation truncation factor 0 HYPRE BoomerAMG: Interpolation: max elements per row 0 HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 HYPRE BoomerAMG: Maximum row sums 0.9 HYPRE BoomerAMG: Sweeps down 1 HYPRE BoomerAMG: Sweeps up 1 HYPRE BoomerAMG: Sweeps on coarse 1 HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax on coarse Gaussian-elimination HYPRE BoomerAMG: Relax weight (all) 1 HYPRE BoomerAMG: Outer relax weight (all) 1 HYPRE BoomerAMG: Using CF-relaxation HYPRE BoomerAMG: Measure type local HYPRE BoomerAMG: Coarsen type HMIS HYPRE BoomerAMG: Interpolation type ext+i linear system matrix = precond matrix: Mat Object: (m_fieldsplit_0_) 24 MPI processes type: mpiaij rows=115812, cols=115812 total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Thanks, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 19 11:20:14 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jul 2019 11:20:14 -0500 Subject: [petsc-users] confused by the converged reason output In-Reply-To: References: Message-ID: On Fri, Jul 19, 2019 at 11:14 AM Michael Wick via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi PETSc team: > > I am a bit confused by the output of converged reason. I set the relative > tolerance to be 10^{-3}. In my run, I get the monitor true residual to be > as follows. > Can you run with -ksp_converged_reason? Thanks, Matt > 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm > 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm > 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 > 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm > 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 > 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm > 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 > > And the converged reason is solve converged due to CONVERGED_RTOL > iterations 3. The relative error is quite far from the prescribed > tolerance. Is there something I should know about the stopping criteria? > > The ks_view output is > Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 > KSP Object:(m_fieldsplit_0_) 24 MPI processes > type: gmres > GMRES: restart=100, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=100, initial guess is zero > tolerances: relative=0.001, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object:(m_fieldsplit_0_) 24 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > HYPRE BoomerAMG: Cycle type V > HYPRE BoomerAMG: Maximum number of levels 25 > HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 > HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 > HYPRE BoomerAMG: Threshold for strong coupling 0.25 > HYPRE BoomerAMG: Interpolation truncation factor 0 > HYPRE BoomerAMG: Interpolation: max elements per row 0 > HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 > HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 > HYPRE BoomerAMG: Maximum row sums 0.9 > HYPRE BoomerAMG: Sweeps down 1 > HYPRE BoomerAMG: Sweeps up 1 > HYPRE BoomerAMG: Sweeps on coarse 1 > HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax on coarse Gaussian-elimination > HYPRE BoomerAMG: Relax weight (all) 1 > HYPRE BoomerAMG: Outer relax weight (all) 1 > HYPRE BoomerAMG: Using CF-relaxation > HYPRE BoomerAMG: Measure type local > HYPRE BoomerAMG: Coarsen type HMIS > HYPRE BoomerAMG: Interpolation type ext+i > linear system matrix = precond matrix: > Mat Object: (m_fieldsplit_0_) 24 MPI processes > type: mpiaij > rows=115812, cols=115812 > total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > Thanks, > > Mike > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.wick.1980 at gmail.com Fri Jul 19 11:21:16 2019 From: michael.wick.1980 at gmail.com (Michael Wick) Date: Fri, 19 Jul 2019 09:21:16 -0700 Subject: [petsc-users] confused by the converged reason output In-Reply-To: References: Message-ID: Yes, it returns: Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 On Fri, Jul 19, 2019 at 9:20 AM Matthew Knepley wrote: > On Fri, Jul 19, 2019 at 11:14 AM Michael Wick via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi PETSc team: >> >> I am a bit confused by the output of converged reason. I set the relative >> tolerance to be 10^{-3}. In my run, I get the monitor true residual to be >> as follows. >> > > Can you run with -ksp_converged_reason? > > Thanks, > > Matt > > >> 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm >> 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm >> 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 >> 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm >> 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 >> 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm >> 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 >> >> And the converged reason is solve converged due to CONVERGED_RTOL >> iterations 3. The relative error is quite far from the prescribed >> tolerance. Is there something I should know about the stopping criteria? >> >> The ks_view output is >> Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 >> KSP Object:(m_fieldsplit_0_) 24 MPI processes >> type: gmres >> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=100, initial guess is zero >> tolerances: relative=0.001, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object:(m_fieldsplit_0_) 24 MPI processes >> type: hypre >> HYPRE BoomerAMG preconditioning >> HYPRE BoomerAMG: Cycle type V >> HYPRE BoomerAMG: Maximum number of levels 25 >> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 >> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 >> HYPRE BoomerAMG: Threshold for strong coupling 0.25 >> HYPRE BoomerAMG: Interpolation truncation factor 0 >> HYPRE BoomerAMG: Interpolation: max elements per row 0 >> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 >> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 >> HYPRE BoomerAMG: Maximum row sums 0.9 >> HYPRE BoomerAMG: Sweeps down 1 >> HYPRE BoomerAMG: Sweeps up 1 >> HYPRE BoomerAMG: Sweeps on coarse 1 >> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination >> HYPRE BoomerAMG: Relax weight (all) 1 >> HYPRE BoomerAMG: Outer relax weight (all) 1 >> HYPRE BoomerAMG: Using CF-relaxation >> HYPRE BoomerAMG: Measure type local >> HYPRE BoomerAMG: Coarsen type HMIS >> HYPRE BoomerAMG: Interpolation type ext+i >> linear system matrix = precond matrix: >> Mat Object: (m_fieldsplit_0_) 24 MPI processes >> type: mpiaij >> rows=115812, cols=115812 >> total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> >> >> Thanks, >> >> Mike >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 19 13:47:04 2019 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jul 2019 13:47:04 -0500 Subject: [petsc-users] confused by the converged reason output In-Reply-To: References: Message-ID: On Fri, Jul 19, 2019 at 11:21 AM Michael Wick wrote: > Yes, it returns: > > Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 > 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 KSP Object:(m_fieldsplit_0_) 24 MPI processes type: gmres GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=100, initial guess is zero tolerances: relative=0.001, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test Note that the relative tolerance is 0.001, so the termination seems correct. Thanks, Matt > On Fri, Jul 19, 2019 at 9:20 AM Matthew Knepley wrote: > >> On Fri, Jul 19, 2019 at 11:14 AM Michael Wick via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hi PETSc team: >>> >>> I am a bit confused by the output of converged reason. I set the >>> relative tolerance to be 10^{-3}. In my run, I get the monitor true >>> residual to be as follows. >>> >> >> Can you run with -ksp_converged_reason? >> >> Thanks, >> >> Matt >> >> >>> 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm >>> 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm >>> 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 >>> 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm >>> 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 >>> 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm >>> 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 >>> >>> And the converged reason is solve converged due to CONVERGED_RTOL >>> iterations 3. The relative error is quite far from the prescribed >>> tolerance. Is there something I should know about the stopping criteria? >>> >>> The ks_view output is >>> Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 >>> KSP Object:(m_fieldsplit_0_) 24 MPI processes >>> type: gmres >>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt >>> Orthogonalization with no iterative refinement >>> GMRES: happy breakdown tolerance 1e-30 >>> maximum iterations=100, initial guess is zero >>> tolerances: relative=0.001, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object:(m_fieldsplit_0_) 24 MPI processes >>> type: hypre >>> HYPRE BoomerAMG preconditioning >>> HYPRE BoomerAMG: Cycle type V >>> HYPRE BoomerAMG: Maximum number of levels 25 >>> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 >>> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 >>> HYPRE BoomerAMG: Threshold for strong coupling 0.25 >>> HYPRE BoomerAMG: Interpolation truncation factor 0 >>> HYPRE BoomerAMG: Interpolation: max elements per row 0 >>> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 >>> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 >>> HYPRE BoomerAMG: Maximum row sums 0.9 >>> HYPRE BoomerAMG: Sweeps down 1 >>> HYPRE BoomerAMG: Sweeps up 1 >>> HYPRE BoomerAMG: Sweeps on coarse 1 >>> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi >>> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi >>> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination >>> HYPRE BoomerAMG: Relax weight (all) 1 >>> HYPRE BoomerAMG: Outer relax weight (all) 1 >>> HYPRE BoomerAMG: Using CF-relaxation >>> HYPRE BoomerAMG: Measure type local >>> HYPRE BoomerAMG: Coarsen type HMIS >>> HYPRE BoomerAMG: Interpolation type ext+i >>> linear system matrix = precond matrix: >>> Mat Object: (m_fieldsplit_0_) 24 MPI processes >>> type: mpiaij >>> rows=115812, cols=115812 >>> total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node (on process 0) routines >>> >>> >>> Thanks, >>> >>> Mike >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.wick.1980 at gmail.com Fri Jul 19 13:57:33 2019 From: michael.wick.1980 at gmail.com (Michael Wick) Date: Fri, 19 Jul 2019 11:57:33 -0700 Subject: [petsc-users] confused by the converged reason output In-Reply-To: References: Message-ID: Hi Matt: At the third iteration, the relative error is 1.313961543155e-02. Am I looking at the correct place? Thanks, M On Fri, Jul 19, 2019 at 11:47 AM Matthew Knepley wrote: > On Fri, Jul 19, 2019 at 11:21 AM Michael Wick > wrote: > >> Yes, it returns: >> >> Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 >> > > 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm > 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm > 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 > 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm > 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 > 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm > 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 > > KSP Object:(m_fieldsplit_0_) 24 MPI processes > type: gmres > GMRES: restart=100, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=100, initial guess is zero > tolerances: relative=0.001, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > > Note that the relative tolerance is 0.001, so the termination seems > correct. > > Thanks, > > Matt > > >> On Fri, Jul 19, 2019 at 9:20 AM Matthew Knepley >> wrote: >> >>> On Fri, Jul 19, 2019 at 11:14 AM Michael Wick via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Hi PETSc team: >>>> >>>> I am a bit confused by the output of converged reason. I set the >>>> relative tolerance to be 10^{-3}. In my run, I get the monitor true >>>> residual to be as follows. >>>> >>> >>> Can you run with -ksp_converged_reason? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm >>>> 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 >>>> 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm >>>> 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 >>>> 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm >>>> 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 >>>> 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm >>>> 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 >>>> >>>> And the converged reason is solve converged due to CONVERGED_RTOL >>>> iterations 3. The relative error is quite far from the prescribed >>>> tolerance. Is there something I should know about the stopping criteria? >>>> >>>> The ks_view output is >>>> Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations >>>> 3 >>>> KSP Object:(m_fieldsplit_0_) 24 MPI processes >>>> type: gmres >>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt >>>> Orthogonalization with no iterative refinement >>>> GMRES: happy breakdown tolerance 1e-30 >>>> maximum iterations=100, initial guess is zero >>>> tolerances: relative=0.001, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object:(m_fieldsplit_0_) 24 MPI processes >>>> type: hypre >>>> HYPRE BoomerAMG preconditioning >>>> HYPRE BoomerAMG: Cycle type V >>>> HYPRE BoomerAMG: Maximum number of levels 25 >>>> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 >>>> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 >>>> HYPRE BoomerAMG: Threshold for strong coupling 0.25 >>>> HYPRE BoomerAMG: Interpolation truncation factor 0 >>>> HYPRE BoomerAMG: Interpolation: max elements per row 0 >>>> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 >>>> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 >>>> HYPRE BoomerAMG: Maximum row sums 0.9 >>>> HYPRE BoomerAMG: Sweeps down 1 >>>> HYPRE BoomerAMG: Sweeps up 1 >>>> HYPRE BoomerAMG: Sweeps on coarse 1 >>>> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi >>>> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi >>>> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination >>>> HYPRE BoomerAMG: Relax weight (all) 1 >>>> HYPRE BoomerAMG: Outer relax weight (all) 1 >>>> HYPRE BoomerAMG: Using CF-relaxation >>>> HYPRE BoomerAMG: Measure type local >>>> HYPRE BoomerAMG: Coarsen type HMIS >>>> HYPRE BoomerAMG: Interpolation type ext+i >>>> linear system matrix = precond matrix: >>>> Mat Object: (m_fieldsplit_0_) 24 MPI processes >>>> type: mpiaij >>>> rows=115812, cols=115812 >>>> total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 >>>> total number of mallocs used during MatSetValues calls =0 >>>> not using I-node (on process 0) routines >>>> >>>> >>>> Thanks, >>>> >>>> Mike >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jul 19 18:59:45 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jul 2019 23:59:45 +0000 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> Message-ID: Thanks, and special thanks for answering every question. In runs did you zero the matrix before call the MatSetValues() initially? > MatSetValues() > FillPetscMat 3.6594e+00 > MatSetValues2_MPIAIJ() with an nrow_buffer = 1 > FillPetscMat 13.3920e+00 > nrow_buffer = 2 > FillPetscMat_with_MatSetValues2 3.3321e+00 > nrow_buffer = 5 > FillPetscMat_with_MatSetValues2 2.8842e+00 > nrow_buffer = 10 > FillPetscMat_with_MatSetValues2 2.7669e+00 > nrow_buffer = 20 > FillPetscMat_with_MatSetValues2 2.6834e+00 > nrow_buffer = 50 > FillPetscMat_with_MatSetValues2 2.6862e+00 > nrow_buffer = 100 > FillPetscMat_with_MatSetValues2 2.6170e+00 > The call to MatSetValues() has a little bit of checking and then another call to MatSetValues_MPIAIJ() so it is not surprising that going directly to MatSetValues_MPIAIJ2() saves you a bit but it is a large savings than I would expect. I am greatly puzzled by the dramatic savings you get as you pass more rows to MatSetValues2. As far as I can see all you are saving is a function call, not much of anything else and that would NOT explain the huge time savings (functions calls are extremely cheap compared to .5 seconds. The multi row MatSetValues2() still has to do the same processing as with one call per row so why so much faster? What kind of a processor is this? Would you be able to run the code under gprof, vtune, instruments or something profiling package that gives line by line information about time being spent. In particular I'd like to see the results for MatSetValues(), MatSetValues2() with 1 row, with 2, rows and with 4 rows. Thanks Barry > On Jul 19, 2019, at 1:34 AM, Mohammed Mostafa wrote: > > 1) Times > a) You run the code that generates the matrix entries and simply comment out the calls to MatSetValues() and it takes > .16747 seconds. > Yes > > b) When you include the MatSetValue calls it takes 3.8820. > Yes > c) When you instead insert into an eigen matrix (which is also parallel?) > No, The Eigen Matrix is just a local matrix to each process and it doesn't support MPI ( at least how I use it ), I just used to establish a baseline for the insertion cost in CSR matrix ( since I don't set off-processor rows anyway) > And it both cases it is a local insertion until MatAssembly where communication and inter-processor copy should happen > After I insert in Eigen and I copy again the Eigen matrix to the Petsc Matrix, using MatUpdateMPIAIJWithArrays() which costs me an additional 0.20326 secs., which is what I intend to use for linear system solution using ksp, pc > it takes 2.8727 secs. > d) When you in your code put all the values into a CSR matrix and then call MatUpdateMPIAIJWithArrays() it takes > 2.8727 secs + .20326 secs. > Yes > e) When you modify MatSetValues_MPIAIJ into MatSetValues_MPIAIJ2() that allows multiple rows that do not need matching > columns it takes 2.6170 secs. > Yes > 1e) When you use MatSetValues_MPIAIJ2() are you directly calling MatSetValues_MPIAIJ2() or are you still calling MatSetValues() and > having it call MatSetValues_MPIAIJ2()? That is did you replace the setvalues() pointer in the function table? > Yes, I am calling MatSetValues2_MPIAIJ() directly not MatSetValues(), I just wanted to confirm the behavior first and if I am getting any performance gain > I wanted to be as minimally intrusive to the code as possible right now to avoid causing problems elsewhere since I have just started learning how it works. > > 1*) For the cases b,c, and e are calling MatSetValues*() multiple times for the same row or exactly once for a given row? > I am only calling MatSetValues exactly once per row and each time I set 4 nonzero values although for the future I may set as many as 7 non-zero values > however, in MatSetValues2_MPIAIJ() I call it once every nrow_buffer > The call is like this > PetscLogEventBegin(FillPetscMatEvent, 0, 0, 0, 0); > auto mat = &PhiEqnSolver.mat; > > > int buffer_size = 100, n_inner, Rows_left,nj_per_row_i; > > > int n_outer = int(ceil(double(nCells) / double(buffer_size))); > > cell > = 0; > > Rows_left > = nCells; > > > for (int i_outer = 0; i_outer < n_outer; i_outer++) { > > n_inner > = min(buffer_size,Rows_left ); > > k > = 0; > > > for (i = 0; i < n_inner; i++) { > > cell_gindex > = cellGIndex[cell]; > > i_index > [i] = cell_gindex; > > > // set the system RHS = f * vol > > Eqn_rhs_ptr > [cell] += cellVol[cell] * rhsFieldptr[cell]; > > > istart > = cellIFace[cell]; > > iend > = cellIFace[cell + 1]; > > > diagCOEF > = 0.0; > > nj_per_row_i > =0; > > > for (fptr = istart; fptr < iend; fptr++) { > > face > = cellFace[fptr]; > > nghbr > = faceNeighbor[face]; > > COEF > = Coef[face]; // flux coef computed on the faces of each cell in the domain > > > if (nghbr < 0) { > > bc_type > = faceBCType[face]; > > > if (bc_type == PROCESSOR) { > > neighbor_cell > = facePhysTag[face]; > > j_index > [k] = neighbor_cell; > > coefvalues > [k] = COEF; > > diagCOEF > -= COEF; > > k > ++; > > nj_per_row_i > ++; > > > } else { > > diagCOEF > -= COEF; > > > } > > > } else { > > owner > = faceOwner[face]; > > neighbor_cell > = owner + nghbr - cell; > > j_index > [k] = cellGIndex[neighbor_cell]; > > coefvalues > [k] = COEF; > > diagCOEF > -= COEF; > > k > ++; > > nj_per_row_i > ++; > > > } > > > } > > j_index > [k] = cell_gindex; > > coefvalues > [k] = diagCOEF; > > k > ++; > > nj_per_row_i > ++; > > nj_perRow > [i] = nj_per_row_i; > > cell > ++; > > Rows_left > --; > > > } > > MatSetValues2_MPIAIJ > (*A, n_inner, i_index, nj_perRow, j_index, coefvalues, INSERT_VALUES); > > > } > > PetscLogEventEnd > (FillPetscMatEvent, 0, 0, 0, 0); > > 2) You have done the initial matrix preallocation with MatCreateMPIAIJWithArrays(); and so have perfect preallocation, that is when you run with > -info and grep for malloc it says 0 mallocs are performed? > Yes no mallocs and no unused non-zeros > check the attached file > > 3) Are you using ADD_VALUES or INSERT_VALUES? > > here are the times for nrow_buffer = 50 > With InsertValues > FillCSTMat_with_MatSetValues 100 1.0 2.5512e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > WithAddValues > FillPetscMat 100 1.0 2.6654e+00 1.0 7.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00 17 53 0 0 0 90100 0 0 0 167 > Additionally I have to zero the matrix which costs > MatZeroEntries 100 1.0 9.5259e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 3 0 0 0 0 0 > INSERT_VALUES is abit faster than ADD_VALUES the difference I saw can be attributed to the cost computation ( addition) > 4) In each MatSetValues() call are you setting values into all nonzero locations of that row, or only a subset of nonzero locations in that row? > Are the column indices for that row you pass in monotonically increasing or not? > Yes I setting new values for all non-zero values in that row, not a subset of it > No, the column indices are Not monotonically increasing as you can see from the above code the column indices come from the mesh connectivity come. > I suppose I can sort them and make line an ordering map for each row and store it and use that to rearrange the values before inserting them. > But would it really matter that much ?! From memory access point, I suppose it could matter. > > 5) Did you ./configure with --with-debugging=0? > PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre > > As for the second Message > 1) Could you please send nrow_buffer = 1 > Yes, Of course. > As I said I made very small changes and now I trying to dig deeper into the code to see if there is more to do > Should I attach the modified < mpiaij.c and petscmat.h > to this message thread or send it directly to you > 2) What time do you get if you use MatSetValues2_MPIAIJ() with an nrow_buffer = 1 ? > Honestly, I didn't try that before. > First I replaced the call in the code above to MatSetValues() and set nrow_buffer = 1 so it should be equivalent to before and here is the time > FillPetscMat 100 1.0 3.6594e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 22 0 0 0 0 96 0 0 0 0 0 > Now I used MatSetValues2_MPIAIJ() with an nrow_buffer = 1 > FillPetscMat 100 1.0 3.3920e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 96 0 0 0 0 0 > > For some reason, MatSetValues2_MPIAIJ is a bit faster. > Sorry for the long reply. > > Thanks, > Kamra > > > On Fri, Jul 19, 2019 at 9:48 AM Smith, Barry F. wrote: > > 1) Could you please send MatSetValues2_MPIAIJ()? > > 2) What time do you get if you use MatSetValues2_MPIAIJ() with an nrow_buffer = 1 ? > > Thanks > > Barry > > > > On Jul 18, 2019, at 2:01 AM, Mohammed Mostafa via petsc-users wrote: > > > > Hello everyone, > > Since I already established a baseline to compare the cost of inserting values in PetscMatrix. > > And based on the hint of the number of values inserted in the matrix each time > > 2) Can you tell me how many values are inserted? > > I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for > > FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same > > > > So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted > > So I made a copy of the function "MatSetValues_MPIAIJ" in "src/mat/impls/aij/mpi/mpiaij.c" and named it "MatSetValues2_MPIAIJ" > > I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices > > > > So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be. > > I tried different number of rows to be inserted i.e. nrow_buffer = [2, 5, 10, 20, 50, 100] > > So now instead of calling "MatSetValues" for every row in the matrix , I call "MatSetValues2_MPIAIJ" every (nrow_buffer)th which should allow for some performance improvement > > the results are as follows > > First Remember that before > > 1-computation and insertion into petsc matrix > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > 2-computation and insertion into eigen matrix > > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > > > Now > > nrow_buffer = 2 > > FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0 > > nrow_buffer = 5 > > FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0 > > nrow_buffer = 10 > > FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 20 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 50 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0 > > nrow_buffer = 100 > > FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 > > > > As to be expected, with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 > > The modifications I made based on MatSetValues_MPIAIJ are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage. > > For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough. > > > > Thanks, > > Kamra > > > > On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa wrote: > > Regarding the first point > > 1) Are you timing only the insertion of values, or computation and insertion? > > I am timing both, the computation and insertion of values but as I said I timed three scenarios > > 1-computation only and no insertion > > Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0 > > 2-computation and insertion into petsc matrix > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > 3-computation and insertion into eigen matrix > > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > I timed 100 times to get a reasonably accurate timings > > > > as for the second point > > 2) Can you tell me how many values are inserted? > > For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376) > > In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row > > the number of off-diagonal onnz in total is 648 nnz > > So I insert nearly 4 values 186062 times ~= 744248 times per mpi process > > > > > > Thanks, > > Kamra > > > > On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley wrote: > > On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa wrote: > > Sorry for the confusion > > First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role. > > So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows > > FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 > > FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 > > > > Great. This helps. Two things would help me narrow down what is happening. > > > > 1) Are you timing only the insertion of values, or computation and insertion? > > > > 2) Can you tell me how many values are inserted? > > > > Thanks, > > > > Matt > > > > I used the same code but simply filled a different Matrix something like > > > > for ( i =0; i < nRows;i++) > > { > > // > > ....... > > // Some code to get j_index, coefvalues > > // Method1 > > MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); > > > > //Method2 > > for ( int k = 0;k < nnz_per_row; k++) > > EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k]; > > > > } > > Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2. > > I found the cost to be trivial in comparison to when either one of the methods is used. > > I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice. > > Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ? > > Maybe some setting or MatOption I should be using so far this what I have been using > > > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE, > > PETSC_DETERMINE, ptr, j , v, A); > > MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE); > > MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); > > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE); > > MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > > MatSetUp(A); > > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > > Thanks, > > Kamra > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > From bsmith at mcs.anl.gov Fri Jul 19 20:42:19 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 20 Jul 2019 01:42:19 +0000 Subject: [petsc-users] confused by the converged reason output In-Reply-To: References: Message-ID: It says right in the -ksp_view left preconditioning using PRECONDITIONED norm type for convergence test The preconditioned residual has decreased by slight more than .001 hence convergence is declared. For GMRES if you want to use the true residual norm decrease for the convergence test you must use -ksp_pc_side right. See KSPSetNormType() and KSPSetPCSide() Barry The reason you can't use the true residual norm with left preconditioner in GMRES is that the GMRES algorithm tracks the preconditioned norm using a recursion formula that is efficient. To calculate the unpreconditioned norm for GMRES with left preconditioning requires forming x_n (expensive) and computing the residual explicitly r_n = b - A x_n (expensive). While the preconditioned residual norm comes with the algorithm for free. > On Jul 19, 2019, at 11:21 AM, Michael Wick via petsc-users wrote: > > Yes, it returns: > > Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 > > > > On Fri, Jul 19, 2019 at 9:20 AM Matthew Knepley wrote: > On Fri, Jul 19, 2019 at 11:14 AM Michael Wick via petsc-users wrote: > Hi PETSc team: > > I am a bit confused by the output of converged reason. I set the relative tolerance to be 10^{-3}. In my run, I get the monitor true residual to be as follows. > > Can you run with -ksp_converged_reason? > > Thanks, > > Matt > > 0 KSP preconditioned resid norm 5.402205955230e-11 true resid norm 9.999870838355e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.240069771831e-12 true resid norm 2.329743436488e-01 ||r(i)||/||b|| 2.329773528226e-01 > 2 KSP preconditioned resid norm 3.394412665922e-13 true resid norm 7.296473323081e-02 ||r(i)||/||b|| 7.296567566748e-02 > 3 KSP preconditioned resid norm 4.936386334724e-14 true resid norm 1.313944571812e-02 ||r(i)||/||b|| 1.313961543155e-02 > > And the converged reason is solve converged due to CONVERGED_RTOL iterations 3. The relative error is quite far from the prescribed tolerance. Is there something I should know about the stopping criteria? > > The ks_view output is > Linear m_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 3 > KSP Object:(m_fieldsplit_0_) 24 MPI processes > type: gmres > GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=100, initial guess is zero > tolerances: relative=0.001, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object:(m_fieldsplit_0_) 24 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > HYPRE BoomerAMG: Cycle type V > HYPRE BoomerAMG: Maximum number of levels 25 > HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 > HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 > HYPRE BoomerAMG: Threshold for strong coupling 0.25 > HYPRE BoomerAMG: Interpolation truncation factor 0 > HYPRE BoomerAMG: Interpolation: max elements per row 0 > HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 > HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 > HYPRE BoomerAMG: Maximum row sums 0.9 > HYPRE BoomerAMG: Sweeps down 1 > HYPRE BoomerAMG: Sweeps up 1 > HYPRE BoomerAMG: Sweeps on coarse 1 > HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax on coarse Gaussian-elimination > HYPRE BoomerAMG: Relax weight (all) 1 > HYPRE BoomerAMG: Outer relax weight (all) 1 > HYPRE BoomerAMG: Using CF-relaxation > HYPRE BoomerAMG: Measure type local > HYPRE BoomerAMG: Coarsen type HMIS > HYPRE BoomerAMG: Interpolation type ext+i > linear system matrix = precond matrix: > Mat Object: (m_fieldsplit_0_) 24 MPI processes > type: mpiaij > rows=115812, cols=115812 > total: nonzeros=1.34986e+06, allocated nonzeros=1.34986e+06 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > Thanks, > > Mike > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From joslorgom at gmail.com Sat Jul 20 05:45:09 2019 From: joslorgom at gmail.com (=?UTF-8?Q?Jos=C3=A9_Lorenzo?=) Date: Sat, 20 Jul 2019 12:45:09 +0200 Subject: [petsc-users] VecGhostRestoreLocalForm Message-ID: Hello, I am not sure I understand the function VecGhostRestoreLocalForm. If I proceed as stated in the manual, VecGhostUpdateBegin (x,INSERT_VALUES ,SCATTER_FORWARD ); VecGhostUpdateEnd (x,INSERT_VALUES ,SCATTER_FORWARD ); VecGhostGetLocalForm (x,&xlocal); VecGetArray (xlocal,&xvalues); // access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; VecRestoreArray (xlocal,&xvalues); VecGhostRestoreLocalForm (x,&xlocal) Does VecRestoreArray update the values in the local vector xlocal, and then VecGhostRestoreLocalForm update the values of the global vector x? Does one need to call these two functions? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Sat Jul 20 08:45:46 2019 From: jczhang at mcs.anl.gov (Zhang, Junchao) Date: Sat, 20 Jul 2019 13:45:46 +0000 Subject: [petsc-users] VecGhostRestoreLocalForm In-Reply-To: References: Message-ID: On Sat, Jul 20, 2019 at 5:47 AM Jos? Lorenzo via petsc-users > wrote: Hello, I am not sure I understand the function VecGhostRestoreLocalForm. If I proceed as stated in the manual, VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD); VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD); VecGhostGetLocalForm(x,&xlocal); VecGetArray(xlocal,&xvalues); // access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; VecRestoreArray(xlocal,&xvalues); VecGhostRestoreLocalForm(x,&xlocal) Does VecRestoreArray update the values in the local vector xlocal, and then VecGhostRestoreLocalForm update the values of the global vector x? Yes, you can think VecRestoreArray finalizes the updates to xlocal. VecGhostRestoreLocalForm does not update global vector. It is for bookkeeping purposes. x and xlocal share the same memory that contains the actual vector data. If you changed ghost points through xvalues[], to get the global vector x updated, you have to call VecGhostUpdateBegin/End after above code, for example, to ADD two ghosts. Does one need to call these two functions? Yes. In PETSc, *Get and *Restore have to be paired. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yyang85 at stanford.edu Sat Jul 20 11:03:21 2019 From: yyang85 at stanford.edu (Yuyun Yang) Date: Sat, 20 Jul 2019 16:03:21 +0000 Subject: [petsc-users] Different behavior of code on different machines Message-ID: Hello team, I'm encountering a problem with my code's behavior on multiple processors. When I run it on my personal computer it works just fine, but when I use it on our computing cluster it produces an error (in one of the root-finding functions, an assert statement is not satisfied) and aborts. If I just run on one processor then both machines can run the code just fine, but they give different results (maybe due to roundoff errors). I'm not sure how to proceed with debugging (since I usually do it on my own computer which didn't seem to encounter a bug) and would appreciate your advice. Thank you! Best regards, Yuyun -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Sat Jul 20 11:13:59 2019 From: jczhang at mcs.anl.gov (Zhang, Junchao) Date: Sat, 20 Jul 2019 16:13:59 +0000 Subject: [petsc-users] Different behavior of code on different machines In-Reply-To: References: Message-ID: You need to test on your personal computer with multiple MPI processes (e.g., mpirun -n 2 ...) before moving to big machines. You may also need to configure petsc with --with-dedugging=1 --COPTFLAGS="-O0 -g" etc to ease debugging. --Junchao Zhang On Sat, Jul 20, 2019 at 11:03 AM Yuyun Yang via petsc-users > wrote: Hello team, I?m encountering a problem with my code?s behavior on multiple processors. When I run it on my personal computer it works just fine, but when I use it on our computing cluster it produces an error (in one of the root-finding functions, an assert statement is not satisfied) and aborts. If I just run on one processor then both machines can run the code just fine, but they give different results (maybe due to roundoff errors). I?m not sure how to proceed with debugging (since I usually do it on my own computer which didn?t seem to encounter a bug) and would appreciate your advice. Thank you! Best regards, Yuyun -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Sat Jul 20 11:47:31 2019 From: jczhang at mcs.anl.gov (Zhang, Junchao) Date: Sat, 20 Jul 2019 16:47:31 +0000 Subject: [petsc-users] Different behavior of code on different machines In-Reply-To: References: Message-ID: Did you used the same number of MPI ranks, same build options on your pc and on cluster? If not, you can try to align options on your pc with those on your cluster to see if you can reproduce the error on your pc. You can also try valgrind to see if there are memory errors like use of uninitialized variables etc. --Junchao Zhang On Sat, Jul 20, 2019 at 11:35 AM Yuyun Yang > wrote: I already tested on my pc with multiple processors and it works fine. I used the command $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 since I configured my PETSc with MPICH, but my local computer has openmpi. Best, Yuyun From: Zhang, Junchao > Sent: Saturday, July 20, 2019 9:14 AM To: Yuyun Yang > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Different behavior of code on different machines You need to test on your personal computer with multiple MPI processes (e.g., mpirun -n 2 ...) before moving to big machines. You may also need to configure petsc with --with-dedugging=1 --COPTFLAGS="-O0 -g" etc to ease debugging. --Junchao Zhang On Sat, Jul 20, 2019 at 11:03 AM Yuyun Yang via petsc-users > wrote: Hello team, I?m encountering a problem with my code?s behavior on multiple processors. When I run it on my personal computer it works just fine, but when I use it on our computing cluster it produces an error (in one of the root-finding functions, an assert statement is not satisfied) and aborts. If I just run on one processor then both machines can run the code just fine, but they give different results (maybe due to roundoff errors). I?m not sure how to proceed with debugging (since I usually do it on my own computer which didn?t seem to encounter a bug) and would appreciate your advice. Thank you! Best regards, Yuyun -------------- next part -------------- An HTML attachment was scrubbed... URL: From mo7ammedmostafa at gmail.com Sat Jul 20 21:58:17 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 21 Jul 2019 11:58:17 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> Message-ID: Hello Barry, > In runs did you zero the matrix before call the MatSetValues() initially? > I only zero the matrix when using add_values insert mode but otherwise I don?t > What kind of a processor is this? This is an i7-Processor 6 core 3.9Ghz if I can remember correctly I tried it on another intel Xeon processor 14 core but using only 6 processor and I got a similar trend with increasing nrow buffer > Would you be able to run the code under gprof, vtune, instruments or something profiling package that gives line by line information about time being spent. In particular I'd like to see the results for MatSetValues(), MatSetValues2() with 1 row, with 2, rows and with 4 rows. > I will try to do that but I am currently traveling to attend a conference so I can not get back to you on that until Thursday Regards, Kamra > On Jul 20, 2019, at 8:59 AM, Smith, Barry F. wrote: > > In runs did you zero the matrix before call the MatSetValues() initially? > > >> MatSetValues() >> FillPetscMat 3.6594e+00 >> MatSetValues2_MPIAIJ() with an nrow_buffer = 1 >> FillPetscMat 13.3920e+00 >> nrow_buffer = 2 >> FillPetscMat_with_MatSetValues2 3.3321e+00 >> nrow_buffer = 5 >> FillPetscMat_with_MatSetValues2 2.8842e+00 >> nrow_buffer = 10 >> FillPetscMat_with_MatSetValues2 2.7669e+00 >> nrow_buffer = 20 >> FillPetscMat_with_MatSetValues2 2.6834e+00 >> nrow_buffer = 50 >> FillPetscMat_with_MatSetValues2 2.6862e+00 >> nrow_buffer = 100 >> FillPetscMat_with_MatSetValues2 2.6170e+00 >> > > The call to MatSetValues() has a little bit of checking and then another call to MatSetValues_MPIAIJ() so it is not surprising that going directly > to MatSetValues_MPIAIJ2() saves you a bit but it is a large savings than I would expect. > > I am greatly puzzled by the dramatic savings you get as you pass more rows to MatSetValues2. As far as I can see all you are saving is a function call, not much of anything else and that would NOT explain the huge time savings (functions calls are extremely cheap compared to .5 seconds. The multi row MatSetValues2() still has to do the same processing as with one call per row so why so much faster? > > What kind of a processor is this? > > Would you be able to run the code under gprof, vtune, instruments or something profiling package that gives line by line information about time being spent. In particular I'd like to see the results for MatSetValues(), MatSetValues2() with 1 row, with 2, rows and with 4 rows. > > > > Thanks > > Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jul 20 22:05:59 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 21 Jul 2019 03:05:59 +0000 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: References: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> Message-ID: <7CFE4386-DF14-4235-B6A6-B4D8476E97A3@mcs.anl.gov> > On Jul 20, 2019, at 9:58 PM, Mohammed Mostafa wrote: > > Hello Barry, > > > In runs did you zero the matrix before call the MatSetValues() initially? > > > I only zero the matrix when using add_values insert mode but otherwise I don?t Is there any reason to use ADD_VALUES? Since you need to insert each value for an individual column/row exactly once? In the timings you sent did you zero the matrix before? Does it make a difference in the timing for setting values into there matrix? Barry > > > > What kind of a processor is this? > This is an i7-Processor 6 core 3.9Ghz if I can remember correctly > I tried it on another intel Xeon processor 14 core but using only 6 processor and I got a similar trend with increasing nrow buffer > > > Would you be able to run the code under gprof, vtune, instruments or something profiling package that gives line by line information about time being spent. In particular I'd like to see the results for MatSetValues(), MatSetValues2() with 1 row, with 2, rows and with 4 rows. > > > I will try to do that but I am currently traveling to attend a conference so I can not get back to you on that until Thursday > > > Regards, > Kamra > > > > On Jul 20, 2019, at 8:59 AM, Smith, Barry F. wrote: > > > > In runs did you zero the matrix before call the MatSetValues() initially? > > > > > >> MatSetValues() > >> FillPetscMat 3.6594e+00 > >> MatSetValues2_MPIAIJ() with an nrow_buffer = 1 > >> FillPetscMat 13.3920e+00 > >> nrow_buffer = 2 > >> FillPetscMat_with_MatSetValues2 3.3321e+00 > >> nrow_buffer = 5 > >> FillPetscMat_with_MatSetValues2 2.8842e+00 > >> nrow_buffer = 10 > >> FillPetscMat_with_MatSetValues2 2.7669e+00 > >> nrow_buffer = 20 > >> FillPetscMat_with_MatSetValues2 2.6834e+00 > >> nrow_buffer = 50 > >> FillPetscMat_with_MatSetValues2 2.6862e+00 > >> nrow_buffer = 100 > >> FillPetscMat_with_MatSetValues2 2.6170e+00 > >> > > > > The call to MatSetValues() has a little bit of checking and then another call to MatSetValues_MPIAIJ() so it is not surprising that going directly > > to MatSetValues_MPIAIJ2() saves you a bit but it is a large savings than I would expect. > > > > I am greatly puzzled by the dramatic savings you get as you pass more rows to MatSetValues2. As far as I can see all you are saving is a function call, not much of anything else and that would NOT explain the huge time savings (functions calls are extremely cheap compared to .5 seconds. The multi row MatSetValues2() still has to do the same processing as with one call per row so why so much faster? > > > > What kind of a processor is this? > > > > Would you be able to run the code under gprof, vtune, instruments or something profiling package that gives line by line information about time being spent. In particular I'd like to see the results for MatSetValues(), MatSetValues2() with 1 row, with 2, rows and with 4 rows. > > > > > > > > Thanks > > > > Barry From mo7ammedmostafa at gmail.com Sun Jul 21 00:11:14 2019 From: mo7ammedmostafa at gmail.com (Mohammed Mostafa) Date: Sun, 21 Jul 2019 14:11:14 +0900 Subject: [petsc-users] Various Questions Regarding PETSC In-Reply-To: <7CFE4386-DF14-4235-B6A6-B4D8476E97A3@mcs.anl.gov> References: <11A7C56B-6787-45E8-A609-27B9D21E7FE2@anl.gov> <7CFE4386-DF14-4235-B6A6-B4D8476E97A3@mcs.anl.gov> Message-ID: Hello > Is there any reason to use ADD_VALUES? Since you need to insert each value for an individual column/row exactly once? > > In the timings you sent did you zero the matrix before? Does it make a difference in the timing for setting values into there matrix? I don?t need to use add values and In all previous emails it is only used once for the sake testing and as a result I use it with MatZeroEntries It costs more time but only very small but MatZeroEntries is what costs some time >> With InsertValues >> FillMAT_with_MatSetValues 100 1.0 2.5512e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0 >> WithAddValues >> FillPetscMat 100 1.0 2.6654e+00 1.0 7.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00 17 53 0 0 0 90100 0 0 0 167 >> Additionally I have to zero the matrix which costs >> MatZeroEntries 100 1.0 9.5259e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 3 0 0 0 0 0 >> INSERT_VALUES is abit faster than ADD_VALUES the difference I saw can be attributed to the cost computation ( addition) Regards, Mohamed M. Kamra > On Jul 21, 2019, at 12:05 PM, Smith, Barry F. wrote: > > Is there any reason to use ADD_VALUES? Since you need to insert each value for an individual column/row exactly once? > > In the timings you sent did you zero the matrix before? Does it make a difference in the timing for setting values into there matrix? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 22 23:58:22 2019 From: jed at jedbrown.org (Jed Brown) Date: Mon, 22 Jul 2019 23:58:22 -0500 Subject: [petsc-users] AGU Session: T003: Advances in Computational Geosciences Message-ID: <87zhl5jqj5.fsf@jedbrown.org> If you are thinking about attending the American Geophysical Union Fall Meeting (Dec 9-13 in San Francisco), please consider submitting an abstract to this interdisciplinary session. Abstracts are due July 31. T003: Advances in Computational Geosciences This session highlights advances in the theory and practice of computational geoscience, from improvements in numerical methods to their application to outstanding problems in the Earth sciences. Common issues include robust and efficient solvers, multiscale discretizations, design of benchmark problems and standards for comparison. Increasing data and computational power necessitates open source scientific libraries and workflow automation for model setup, 3D feature connectivity, and data assimilation, and automation in uncertainty representation and propagation, optimal design of field studies, risk quantification, and testing the predictive power of numerical simulations. By bringing these crosscutting computational activities together in one session, we hope to sharpen our collective understanding of fundamental challenges, level of rigor, and opportunities for reusable implementations. Contributions from all areas are welcome, including, but not limited to, fault modeling, tectonics, subduction, seismology, magma dynamics, mantle convection, the core, as well as surface processes, hydrology, and cryosphere. Confirmed invited presenters: Talea Mayo, Andreas Fichtner https://agu.confex.com/agu/fm19/prelim.cgi/Session/83797 Conveners Jed Brown University of Colorado at Boulder Alice-Agnes Gabriel Ludwig-Maximilians-Universit?t Georg S Reuber Johannes Gutenberg University of Mainz Nathan Collier Oak Ridge National Laboratory From thibault.bridelbertomeu at gmail.com Tue Jul 23 01:24:59 2019 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Tue, 23 Jul 2019 08:24:59 +0200 Subject: [petsc-users] How to stop a TS early In-Reply-To: References: Message-ID: Thank you very much for the info Lisandro. Thibault Bridel-Bertomeu Le ven. 19 juil. 2019 ? 17:33, Lisandro Dalcin a ?crit : > > PetscErrorCode YourPostStepCallback(...) { > ... > ierr = TSSetConvergedReason(ts,TS_CONVERGED_USER);CHKERRQ(ierr); > ... > } > > On Fri, 19 Jul 2019 at 18:21, Thibault Bridel-Bertomeu via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Petsc users, >> >> I am facing the following situation. >> I run a code that uses a Petsc TimeStepper on a supercomputer that has a >> given walltime. In real time, I am able to get the remaining time, and when >> I have only a few minutes left I want to stop the code in a clean fashion, >> rather than an abrupt out-of-time. I set-up all the machinery to check the >> remaining time already in the TSPostStep so the code checks after each >> iteration, and sets a flag to 1 when it got to stop. >> >> Now what I would like to know, is how to tell TSSolve to stop iterating >> when that flag is set to 1 in the TSPostStep ? >> >> Any clues would be much appreciated ! >> >> Best regards, >> >> Thibault Bridel-Bertomeu >> > > > -- > Lisandro Dalcin > ============ > Research Scientist > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barrydog505 at gmail.com Tue Jul 23 02:43:49 2019 From: barrydog505 at gmail.com (Tsung-Hsing Chen) Date: Tue, 23 Jul 2019 15:43:49 +0800 Subject: [petsc-users] Get stiffness matrix by using Petsc's functions Message-ID: Hi, I'am using Petsc to solve FEM problems. Right now I just use MatSetValue() or MatSetValues() to build up the stiffness matrix. I wonder can I easily get stiffness matrix by using Petsc's functions. By the way, I'am using unstructured grid (DMPlex). Thanks a lot for your help, Tsung-Hsing Chen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 23 07:53:59 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Jul 2019 07:53:59 -0500 Subject: [petsc-users] Get stiffness matrix by using Petsc's functions In-Reply-To: References: Message-ID: On Tue, Jul 23, 2019 at 2:45 AM Tsung-Hsing Chen via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi, > I'am using Petsc to solve FEM problems. > Right now I just use MatSetValue() or MatSetValues() to build up the > stiffness matrix. > I wonder can I easily get stiffness matrix by using Petsc's functions. > By the way, I'am using unstructured grid (DMPlex). > PETSc has some support to generate very simple FEM problems. Almost everything we can do is shown in examples. For instance, SNES ex12: Poisson SNES ex17: Elasticity SNES ex62: Stokes SNES ex69: nonlinear Stokes SNES ex77: large deformation Elasticity TS ex45: Heat equation TS ex46: Navier-Stokes If those look understandable, then maybe this system is capable of generating your problem. Thanks, Matt Thanks a lot for your help, > Tsung-Hsing Chen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barrydog505 at gmail.com Tue Jul 23 08:02:33 2019 From: barrydog505 at gmail.com (Tsung-Hsing Chen) Date: Tue, 23 Jul 2019 21:02:33 +0800 Subject: [petsc-users] Get stiffness matrix by using Petsc's functions In-Reply-To: References: Message-ID: I'll check them out. Thanks, Tsung-Hsing Chen Matthew Knepley ? 2019?7?23? ?? ??8:54??? > On Tue, Jul 23, 2019 at 2:45 AM Tsung-Hsing Chen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi, >> I'am using Petsc to solve FEM problems. >> Right now I just use MatSetValue() or MatSetValues() to build up the >> stiffness matrix. >> I wonder can I easily get stiffness matrix by using Petsc's functions. >> By the way, I'am using unstructured grid (DMPlex). >> > > PETSc has some support to generate very simple FEM problems. Almost > everything we can do is shown > in examples. For instance, > > SNES ex12: Poisson > SNES ex17: Elasticity > SNES ex62: Stokes > SNES ex69: nonlinear Stokes > SNES ex77: large deformation Elasticity > TS ex45: Heat equation > TS ex46: Navier-Stokes > > If those look understandable, then maybe this system is capable of > generating your problem. > > Thanks, > > Matt > > Thanks a lot for your help, >> Tsung-Hsing Chen >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.appel17 at imperial.ac.uk Tue Jul 23 09:24:14 2019 From: t.appel17 at imperial.ac.uk (Thibaut Appel) Date: Tue, 23 Jul 2019 15:24:14 +0100 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS Message-ID: Dear PETSc users, I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. Would you support such a feature? Thanks for your support, Thibaut From epscodes at gmail.com Tue Jul 23 09:29:32 2019 From: epscodes at gmail.com (Xiangdong) Date: Tue, 23 Jul 2019 10:29:32 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: Hello everyone, I have two quick questions related to run gpu solvers. 1) # of MPI processes vs # of GPUs. Is it true that we should set these two numbers equal if most of computations are done on GPU? For one case I tested, with only one GPU, running with np=2 is 15% slower than np=1 (probably due to mpi communications). I am curious in what case, one will benefit by having more mpi processes than number of GPUs. 2) When I use MatLoad(A,viewer) to load a binary format data into a aijcusparse format A, how many matrices are created? Does it involve creating a intermediate aij matrix A' on CPU then convert it to aijcuspare A on GPU? I am not sure whether such intermediate aij matrix exist. If yes, What is the life time of this matrix? Is it destroyed right after the converting? Thanks for your help. Best, Xiangdong On Fri, Jul 19, 2019 at 12:08 PM Karl Rupp wrote: > Hi Xiangdong, > > > > I can understand some of the numbers, but not the HtoD case. > > > > In DtoH1, it is the data movement from VecMDot. The size of data is > > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = 8*128*8 > > = 8192. My question is: instead of calling cublasDdot nv times, why do > > you implement your own kernels? I guess it must be for performance, but > > can you explain a little more? > > Yes, this is a performance optimization. We've used several dot-products > (suffers from kernel launch latency) as well as matrix-vector-products > (suffers extra matrix setup) in the past; in both cases, there was extra > memory traffic, thus impacting performance. > > The reason why the data size is 8192 is to get around a separate > reduction stage on the GPU (i.e. a second kernel launch). By moving the > data to the CPU and doing the reduction there, one is faster than doing > it on the GPU and then moving only a few numbers. This has to do with > PCI-Express latency: It takes about the same time to send a single byte > as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes the > limiting factor. > > > > > In DtoH2, it is the data movement from VecNorm. The size of data is 8B, > > which is just the sizeof(PetscScalar). > > This is most likely the result required for the control flow on the CPU. > > > > In DtoD1, it is the data movement from VecAXPY. The size of data is > > 17.952MB, which is exactly sizeof(PetscScalar)*length(b). > > This is a vector assignment. If I remember correctly, it uses the > memcpy-routines and hence shows up as a separate DtoD instead of just a > kernel. It matches the time required for scal_kernel_val (scaling a > vector by a scalar), so it runs at full bandwidth on the GPU. > > > > However, I do not understand the number in HostToDevice in gmres for > > np=1. The size of data movement is 1.032KB. I thought this is related to > > the updated upper Hessenberg matrix, but the number does not match. Can > > anyone help me understand the data movement of HToD in GMRES for np=1? > > 1032 = (128+1)*8, so this might be some auxiliary work information on > the GPU. I could figure out the exact source of these transfers, but > that is some effort. Let me know whether this is important information > for you, then I can do it. > > Best regards, > Karli > > > > > > > > Thank you. > > > > Best, > > Xiangdong > > > > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp > > wrote: > > > > Hi, > > > > as you can see from the screenshot, the communication is merely for > > scalars from the dot-products and/or norms. These are needed on the > > host > > for the control flow and convergence checks and is true for any > > iterative solver. > > > > Best regards, > > Karli > > > > > > > > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > > > > > > > > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > > > > > >> wrote: > > > > > > > > > 1) What preconditioner are you using? If any. > > > > > > Currently I am using none as I want to understand how gmres works > > on GPU. > > > > > > > > > 2) Where/how are you getting this information about the > > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like to > utilize > > > this same sort of information to plan future optimizations. > > > > > > I am using nvprof and nvvp from cuda toolkit. It looks like there > > are > > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per iteration for > np=1 > > > case. See the attached snapshots. > > > > > > 3) Are you using more than 1 MPI rank? > > > > > > > > > I tried both np=1 and np=2. Attached please find snapshots from > > nvvp for > > > both np=1 and np=2 cases. The figures showing gpu calls with two > > pure > > > gmres iterations. > > > > > > Thanks. > > > Xiangdong > > > > > > > > > If you use the master branch (which we highly recommend for > > > anyone using GPUs and PETSc) the -log_view option will log > > > communication between CPU and GPU and display it in the > summary > > > table. This is useful for seeing exactly what operations are > > doing > > > vector communication between the CPU/GPU. > > > > > > We welcome all feedback on the GPUs since it previously > > has only > > > been lightly used. > > > > > > Barry > > > > > > > > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > > > > > >> > > wrote: > > > > > > > > Hello everyone, > > > > > > > > I am new to petsc gpu and have a simple question. > > > > > > > > When I tried to solve Ax=b where A is MATAIJCUSPARSE and b > > and x > > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found that > > during > > > each krylov iteration, there are one call MemCpy(HtoD) and > > one call > > > MemCpy(DtoH). Does that mean the Krylov solve is not 100% on > > GPU and > > > the solve still needs some work from CPU? What are these > > MemCpys for > > > during the each iteration? > > > > > > > > Thank you. > > > > > > > > Best, > > > > Xiangdong > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Jul 23 11:02:42 2019 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Tue, 23 Jul 2019 16:02:42 +0000 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: Message-ID: Thibaut : How large is your A_m? Write/Read to/from the disc is non-scalable. It would be the last resort when matrix factor is too large for the memory. PETSc/mumps interface does not support this feature. If A_m is not huge, you can create one KSP, with matrix A = diag(A_m). Keep LU factor of A = diag( LU factor of A_m), then solve A x = b repeatedly with changing b. You can use '-pc_type bjacobi -pc_bjacobi_blocks M -sub_pc_type lu' Hong Dear PETSc users, I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. Would you support such a feature? Thanks for your support, Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 23 11:13:53 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 23 Jul 2019 16:13:53 +0000 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: Message-ID: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> What types of computing systems will you be doing the computations? Roughly how many MPI_ranks? Are the matrices all the same size? Do they have the same or different nonzero structures? Would it be possible to use the same symbolic representation for all of them and just have different numerical values? Clusters and large scale computing centers are notoriously terrible at IO; often IO is orders of magnitude slower than compute/memory making this type of workflow unrealistically slow. From a cost analysis point of view often just buying lots of memory might be the most efficacious approach. That said, what you suggest might require only a few lines of code (though determining where to put them is the tricky part) depending on the MUMPS interface for saving a filer to disk. What we would do is keep the PETSc wrapper that lives around the MUMPS matrix Mat_MUMPS but using the MUMPS API save the information in the DMUMPS_STRUC_C id; and then reload it when needed. The user level API could be something like MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they would just money with DMUMPS_STRUC_C id; item. Barry > On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users wrote: > > Dear PETSc users, > > I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. > > The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. > The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. > > The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. > Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. > > In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. > > I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. > > Would you support such a feature? > > Thanks for your support, > > Thibaut From t.appel17 at imperial.ac.uk Tue Jul 23 11:13:47 2019 From: t.appel17 at imperial.ac.uk (Thibaut Appel) Date: Tue, 23 Jul 2019 17:13:47 +0100 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: Message-ID: <9164193c-9d4d-1573-8888-ae36542ff100@imperial.ac.uk> Hi Hong, A_m would typically have a leading dimension between 6e5 and 1.5e6, with roughly 100 non-zero entries per row in average. Don't get me wrong: performing ONE LU factorization is fine for the memory. It's just that I need to keep track, and store M x LU factorizations which obviously do not fit in RAM. I thought that with MUMPS, each process is able to store its part of the LU factors? Anyway, I do not care about the non-scalability, I know saving/reading from disk is long. As long as I solve the systems in parallel it's fine. > If A_m is not huge, you can create one KSP, with matrix > A = diag(A_m). > Keep LU factor of A = diag( LU factor of A_m), then solve A x = b > repeatedly with changing b. > You can use '-pc_type bjacobi -pc_bjacobi_blocks M -sub_pc_type lu' > Hong > This is likely to fail as my matrices are ill-conditioned. Thibaut On 23/07/2019 17:02, Zhang, Hong wrote: > Thibaut : > How large is your A_m? Write/Read to/from the disc is non-scalable. It > would be the last resort when matrix factor is too large for the > memory. PETSc/mumps interface does not support this feature. > > If A_m is not huge, you can create one KSP, with matrix > A = diag(A_m). > Keep LU factor of A = diag( LU factor of A_m), then solve A x = b > repeatedly with changing b. > You can use '-pc_type bjacobi -pc_bjacobi_blocks M -sub_pc_type lu' > Hong > > Dear PETSc users, > > I need to solve several linear systems successively, with LU > factorization, as part of an iterative process in my Fortran > application > code. > > The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at > each iteration i, but computing the LU factorization of A_m only once. > The RHSs (b_m,i+1) are computed from all the different (x_m,i) and > all > depend upon each other. > > The way I envisage to perform that is to use MUMPS to compute, > successively, each of the LU factorizations (m) in parallel and store > the factors on disk, creating/assembling/destroying the matrices > A_m on > the go. > Then whenever needed, read the factors in parallel to solve the > systems. > Since version 5.2, MUMPS has a save/restore feature that allows that, > see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf > p.20, 24 and 58. > > In its current state, the PETSc/MUMPS interface does not incorporate > that feature. I'm an advanced Fortran programmer but not in C so I > don't > think I would do an amazing job having a go inside > src/mat/impls/aij/mpi/mumps/mumps.c. > > I was picturing something like creating as many KSP objects as linear > systems to be solved, with some sort of flag to force the storage > of LU > factors on disk after the first call to KSPSolve. Then keep calling > KSPSolve as many times as needed. > > Would you support such a feature? > > Thanks for your support, > > Thibaut > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.appel17 at imperial.ac.uk Tue Jul 23 11:31:32 2019 From: t.appel17 at imperial.ac.uk (Thibaut Appel) Date: Tue, 23 Jul 2019 17:31:32 +0100 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> Message-ID: Hi Barry, Thanks for taking the time. I would typically run that on a small cluster node of 16 or 32 physical cores with 2 or 4 sockets. I use 16 or 32 MPI ranks and bind them to cores. The matrices would ALL have the same size and the same nonzero structure - it's just a few numerical values that would differ. This is a good point you've raised as I don't think MUMPS is able to exploit that - I asked the question in their users list just to be sure. There are some options in SuperLU dist to reuse permutation arrays, but there's no I/O for that solver. And the native PETSc LU solver is not parallel? I'm using high-order finite differences so I'm suffering from a lot of fill-in, one of the reasons why storing factorizations in RAM is not viable. In comparison, I have almost unlimited disk space. I'm aware my need might seem counter-intuitive, but I'm really willing to sacrifice performance in the I/O part. My code is already heavily based on PETSc (preallocation, assembly for matrices/vectors) coupled with MUMPS I'm minimizing the loss of efficiency. Thibaut On 23/07/2019 17:13, Smith, Barry F. wrote: > What types of computing systems will you be doing the computations? Roughly how many MPI_ranks? > > Are the matrices all the same size? Do they have the same or different nonzero structures? Would it be possible to use the same symbolic representation for all of them and just have different numerical values? > > Clusters and large scale computing centers are notoriously terrible at IO; often IO is orders of magnitude slower than compute/memory making this type of workflow unrealistically slow. From a cost analysis point of view often just buying lots of memory might be the most efficacious approach. > > That said, what you suggest might require only a few lines of code (though determining where to put them is the tricky part) depending on the MUMPS interface for saving a filer to disk. What we would do is keep the PETSc wrapper that lives around the MUMPS matrix Mat_MUMPS but using the MUMPS API save the information in the DMUMPS_STRUC_C id; and then reload it when needed. > > The user level API could be something like > > MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they would just money with DMUMPS_STRUC_C id; item. > > > Barry > > >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users wrote: >> >> Dear PETSc users, >> >> I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. >> >> The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. >> The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. >> >> The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. >> Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. >> >> In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. >> >> I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. >> >> Would you support such a feature? >> >> Thanks for your support, >> >> Thibaut From rupp at iue.tuwien.ac.at Tue Jul 23 12:24:56 2019 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Tue, 23 Jul 2019 19:24:56 +0200 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: Hi, > I have two quick questions related to run gpu solvers. > > 1) # of MPI processes vs # of GPUs. Is it true that we should set these > two numbers equal if most of computations are done on GPU? For one case > I tested, with only one GPU, running with np=2 is 15% slower than np=1 > (probably due to mpi communications). I am curious in what case, one > will benefit by having more mpi processes than number of GPUs. yes, usually you want to have the same number of MPI processes as the number of GPUs on each node, unless you have a lot of work for the CPU and only little work for the GPU. If you oversubscribe the GPU, it will not get any faster (unlike multi-core CPUs); instead, the extra management overhead will make it slow down. > 2) When I use MatLoad(A,viewer) to load a binary format data into a > aijcusparse format A, how many matrices are created? Does it involve > creating a intermediate aij matrix A' on CPU then convert it to > aijcuspare A on GPU? I am not sure whether such intermediate aij matrix > exist. If yes, What is the life time of this matrix? Is it destroyed > right after the converting? GPU matrices are essentially AIJ matrices with additional GPU data members. That is, if you MatLoad(), the data will be copied into the CPU buffers first and then pushed down to the GPU when needed. The CPU buffers will never be free'd, but might be updated from the latest GPU data to allow for fallback operations that have no GPU implementation. Best regards, Karli > On Fri, Jul 19, 2019 at 12:08 PM Karl Rupp > wrote: > > Hi Xiangdong, > > > > I can understand some of the numbers, but not the HtoD case. > > > > In DtoH1, it is the data movement from VecMDot. The size of data is > > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = > 8*128*8 > > = 8192. My question is: instead of calling cublasDdot nv times, > why do > > you implement your own kernels? I guess it must be for > performance, but > > can you explain a little more? > > Yes, this is a performance optimization. We've used several > dot-products > (suffers from kernel launch latency) as well as matrix-vector-products > (suffers extra matrix setup) in the past; in both cases, there was > extra > memory traffic, thus impacting performance. > > The reason why the data size is 8192 is to get around a separate > reduction stage on the GPU (i.e. a second kernel launch). By moving the > data to the CPU and doing the reduction there, one is faster than doing > it on the GPU and then moving only a few numbers. This has to do with > PCI-Express latency: It takes about the same time to send a single byte > as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes > the > limiting factor. > > > > > In DtoH2, it is the data movement from VecNorm. The size of data > is 8B, > > which is just the sizeof(PetscScalar). > > This is most likely the result required for the control flow on the CPU. > > > > In DtoD1, it is the data movement from VecAXPY. The size of data is > > 17.952MB, which is exactly sizeof(PetscScalar)*length(b). > > This is a vector assignment. If I remember correctly, it uses the > memcpy-routines and hence shows up as a separate DtoD instead of just a > kernel. It matches the time required for scal_kernel_val (scaling a > vector by a scalar), so it runs at full bandwidth on the GPU. > > > > However, I do not understand the number in HostToDevice in gmres for > > np=1. The size of data movement is 1.032KB. I thought this is > related to > > the updated upper Hessenberg matrix, but the number does not > match. Can > > anyone help me understand the data movement of HToD in GMRES for > np=1? > > 1032 = (128+1)*8, so this might be some auxiliary work information on > the GPU. I could figure out the exact source of these transfers, but > that is some effort. Let me know whether this is important information > for you, then I can do it. > > Best regards, > Karli > > > > > > > > Thank you. > > > > Best, > > Xiangdong > > > > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp > > >> wrote: > > > >? ? ?Hi, > > > >? ? ?as you can see from the screenshot, the communication is > merely for > >? ? ?scalars from the dot-products and/or norms. These are needed > on the > >? ? ?host > >? ? ?for the control flow and convergence checks and is true for any > >? ? ?iterative solver. > > > >? ? ?Best regards, > >? ? ?Karli > > > > > > > >? ? ?On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > >? ? ? > > >? ? ? > > >? ? ? > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > >? ? ? > > > >? ? ? > > >>> wrote: > >? ? ? > > >? ? ? > > >? ? ? >? ? ? ? ?1) What preconditioner are you using? If any. > >? ? ? > > >? ? ? > Currently I am using none as I want to understand how > gmres works > >? ? ?on GPU. > >? ? ? > > >? ? ? > > >? ? ? >? ? ? ? ?2) Where/how are you getting this information > about the > >? ? ? >? ? ?MemCpy(HtoD) and one call MemCpy(DtoH)? We might like > to utilize > >? ? ? >? ? ?this same sort of information to plan future > optimizations. > >? ? ? > > >? ? ? > I am using nvprof and nvvp from cuda toolkit. It looks > like there > >? ? ?are > >? ? ? > one MemCpy(HtoD) and three MemCpy(DtoH) calls per > iteration for np=1 > >? ? ? > case. See the attached snapshots. > >? ? ? > > >? ? ? >? ? ? ? ?3) Are you using more than 1 MPI rank? > >? ? ? > > >? ? ? > > >? ? ? > I tried both np=1 and np=2. Attached please find snapshots > from > >? ? ?nvvp for > >? ? ? > both np=1 and np=2 cases. The figures showing gpu calls > with two > >? ? ?pure > >? ? ? > gmres iterations. > >? ? ? > > >? ? ? > Thanks. > >? ? ? > Xiangdong > >? ? ? > > >? ? ? > > >? ? ? >? ? ? ? If you use the master branch (which we highly > recommend for > >? ? ? >? ? ?anyone using GPUs and PETSc) the -log_view option will log > >? ? ? >? ? ?communication between CPU and GPU and display it in > the summary > >? ? ? >? ? ?table. This is useful for seeing exactly what > operations are > >? ? ?doing > >? ? ? >? ? ?vector communication between the CPU/GPU. > >? ? ? > > >? ? ? >? ? ? ? We welcome all feedback on the GPUs since it previously > >? ? ?has only > >? ? ? >? ? ?been lightly used. > >? ? ? > > >? ? ? >? ? ? ? ?Barry > >? ? ? > > >? ? ? > > >? ? ? >? ? ? > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users > >? ? ? >? ? ? > > >? ? ? >>> > >? ? ?wrote: > >? ? ? >? ? ? > > >? ? ? >? ? ? > Hello everyone, > >? ? ? >? ? ? > > >? ? ? >? ? ? > I am new to petsc gpu and have a simple question. > >? ? ? >? ? ? > > >? ? ? >? ? ? > When I tried to solve Ax=b where A is > MATAIJCUSPARSE and b > >? ? ?and x > >? ? ? >? ? ?are VECSEQCUDA? with GMRES(or GCR) and pcnone, I found > that > >? ? ?during > >? ? ? >? ? ?each krylov iteration, there are one call MemCpy(HtoD) and > >? ? ?one call > >? ? ? >? ? ?MemCpy(DtoH). Does that mean the Krylov solve is not > 100% on > >? ? ?GPU and > >? ? ? >? ? ?the solve still needs some work from CPU? What are these > >? ? ?MemCpys for > >? ? ? >? ? ?during the each iteration? > >? ? ? >? ? ? > > >? ? ? >? ? ? > Thank you. > >? ? ? >? ? ? > > >? ? ? >? ? ? > Best, > >? ? ? >? ? ? > Xiangdong > >? ? ? > > > > From hzhang at mcs.anl.gov Tue Jul 23 12:38:10 2019 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Tue, 23 Jul 2019 17:38:10 +0000 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> Message-ID: Thibaut: Thanks for taking the time. I would typically run that on a small cluster node of 16 or 32 physical cores with 2 or 4 sockets. I use 16 or 32 MPI ranks and bind them to cores. The matrices would ALL have the same size and the same nonzero structure - it's just a few numerical values that would differ. You may do one symbolic factorization of A_m, use it in the m-i loop: - numeric factorization of A_m - solve A_m x_m,i = b_m,i in mumps, numeric factorization and solve are scalable. Repeated numeric factorization of A_m are likely faster than reading data files from the disc. Hong This is a good point you've raised as I don't think MUMPS is able to exploit that - I asked the question in their users list just to be sure. There are some options in SuperLU dist to reuse permutation arrays, but there's no I/O for that solver. And the native PETSc LU solver is not parallel? I'm using high-order finite differences so I'm suffering from a lot of fill-in, one of the reasons why storing factorizations in RAM is not viable. In comparison, I have almost unlimited disk space. I'm aware my need might seem counter-intuitive, but I'm really willing to sacrifice performance in the I/O part. My code is already heavily based on PETSc (preallocation, assembly for matrices/vectors) coupled with MUMPS I'm minimizing the loss of efficiency. Thibaut On 23/07/2019 17:13, Smith, Barry F. wrote: > What types of computing systems will you be doing the computations? Roughly how many MPI_ranks? > > Are the matrices all the same size? Do they have the same or different nonzero structures? Would it be possible to use the same symbolic representation for all of them and just have different numerical values? > > Clusters and large scale computing centers are notoriously terrible at IO; often IO is orders of magnitude slower than compute/memory making this type of workflow unrealistically slow. From a cost analysis point of view often just buying lots of memory might be the most efficacious approach. > > That said, what you suggest might require only a few lines of code (though determining where to put them is the tricky part) depending on the MUMPS interface for saving a filer to disk. What we would do is keep the PETSc wrapper that lives around the MUMPS matrix Mat_MUMPS but using the MUMPS API save the information in the DMUMPS_STRUC_C id; and then reload it when needed. > > The user level API could be something like > > MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they would just money with DMUMPS_STRUC_C id; item. > > > Barry > > >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users > wrote: >> >> Dear PETSc users, >> >> I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. >> >> The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. >> The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. >> >> The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. >> Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. >> >> In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. >> >> I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. >> >> Would you support such a feature? >> >> Thanks for your support, >> >> Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Tue Jul 23 14:17:03 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Tue, 23 Jul 2019 12:17:03 -0700 Subject: [petsc-users] Low floating precision numerical solver? Message-ID: Hello, I was wondering if PETSc had some form of a low precision linear solver algorithm like seen in: https://www.dropbox.com/s/rv5quc3k72qdpmp/iciam_lowprec19.pdf?dl=0 I understand this treatment is coming from one of the NAG library developers, Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 23 15:00:46 2019 From: jed at jedbrown.org (Jed Brown) Date: Tue, 23 Jul 2019 15:00:46 -0500 Subject: [petsc-users] Low floating precision numerical solver? In-Reply-To: References: Message-ID: <87o91kikr5.fsf@jedbrown.org> You can use reduced-precision preconditioning if you're writing your own, but there isn't out-of-the-box support. Note that the benefit is limited when working with sparse matrices because a lot of the cost comes from memory access (including column indices) and vectorization for some operations is difficult. Manuel Valera via petsc-users writes: > Hello, > > I was wondering if PETSc had some form of a low precision linear solver > algorithm like seen in: > > https://www.dropbox.com/s/rv5quc3k72qdpmp/iciam_lowprec19.pdf?dl=0 > > I understand this treatment is coming from one of the NAG library > developers, > > Thanks, From bsmith at mcs.anl.gov Tue Jul 23 15:01:29 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 23 Jul 2019 20:01:29 +0000 Subject: [petsc-users] Low floating precision numerical solver? In-Reply-To: References: Message-ID: Manuel, We don't have support for this yet for GPUs, it is on our list but not a priority, just utilizing GPUs well is our first priority. Barry Note what the people who give these types of presentations forget to emphasis enough is that though this techniques for some algorithms can give good benefits on GPUs utilizing lower precision on CPUs doesn't generally benefit you much. > On Jul 23, 2019, at 2:17 PM, Manuel Valera via petsc-users wrote: > > Hello, > > I was wondering if PETSc had some form of a low precision linear solver algorithm like seen in: > > https://www.dropbox.com/s/rv5quc3k72qdpmp/iciam_lowprec19.pdf?dl=0 > > I understand this treatment is coming from one of the NAG library developers, > > Thanks, > > From mvalera-w at sdsu.edu Tue Jul 23 15:02:35 2019 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Tue, 23 Jul 2019 13:02:35 -0700 Subject: [petsc-users] Low floating precision numerical solver? In-Reply-To: References: Message-ID: Understood, thanks for your answers, On Tue, Jul 23, 2019 at 1:01 PM Smith, Barry F. wrote: > > Manuel, > > We don't have support for this yet for GPUs, it is on our list but not > a priority, just utilizing GPUs well is our first priority. > > Barry > > > Note what the people who give these types of presentations forget to > emphasis enough is that though this techniques for some algorithms can give > good benefits on GPUs utilizing lower precision on CPUs doesn't generally > benefit you much. > > > > On Jul 23, 2019, at 2:17 PM, Manuel Valera via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hello, > > > > I was wondering if PETSc had some form of a low precision linear solver > algorithm like seen in: > > > > https://www.dropbox.com/s/rv5quc3k72qdpmp/iciam_lowprec19.pdf?dl=0 > > > > I understand this treatment is coming from one of the NAG library > developers, > > > > Thanks, > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 23 17:50:47 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 23 Jul 2019 22:50:47 +0000 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> Message-ID: <793A6A41-FAC1-4E61-9905-539B18BEE392@mcs.anl.gov> > On Jul 23, 2019, at 12:24 PM, Karl Rupp wrote: > > Hi, > >> I have two quick questions related to run gpu solvers. >> 1) # of MPI processes vs # of GPUs. Is it true that we should set these two numbers equal if most of computations are done on GPU? For one case I tested, with only one GPU, running with np=2 is 15% slower than np=1 (probably due to mpi communications). I am curious in what case, one will benefit by having more mpi processes than number of GPUs. > > yes, usually you want to have the same number of MPI processes as the number of GPUs on each node, unless you have a lot of work for the CPU and only little work for the GPU. If you oversubscribe the GPU, it will not get any faster (unlike multi-core CPUs); instead, the extra management overhead will make it slow down. > > >> 2) When I use MatLoad(A,viewer) to load a binary format data into a aijcusparse format A, how many matrices are created? Does it involve creating a intermediate aij matrix A' on CPU then convert it to aijcuspare A on GPU? I am not sure whether such intermediate aij matrix exist. If yes, What is the life time of this matrix? Is it destroyed right after the converting? > > GPU matrices are essentially AIJ matrices with additional GPU data members. That is, if you MatLoad(), the data will be copied into the CPU buffers first and then pushed down to the GPU when needed. The CPU buffers will never be free'd, but might be updated from the latest GPU data to allow for fallback operations that have no GPU implementation. Eventually we should provide a mechanism that allows users to free the CPU side of matrices (and vectors) if the user knows that they will never need them again on the CPU. Honestly we are still working on the basics, not yet at this level of optimization. If you would benefit from such a feature now feel free to make a pull request. Barry > > Best regards, > Karli > > > > > >> On Fri, Jul 19, 2019 at 12:08 PM Karl Rupp > wrote: >> Hi Xiangdong, >> > I can understand some of the numbers, but not the HtoD case. >> > >> > In DtoH1, it is the data movement from VecMDot. The size of data is >> > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = >> 8*128*8 >> > = 8192. My question is: instead of calling cublasDdot nv times, >> why do >> > you implement your own kernels? I guess it must be for >> performance, but >> > can you explain a little more? >> Yes, this is a performance optimization. We've used several >> dot-products >> (suffers from kernel launch latency) as well as matrix-vector-products >> (suffers extra matrix setup) in the past; in both cases, there was >> extra >> memory traffic, thus impacting performance. >> The reason why the data size is 8192 is to get around a separate >> reduction stage on the GPU (i.e. a second kernel launch). By moving the >> data to the CPU and doing the reduction there, one is faster than doing >> it on the GPU and then moving only a few numbers. This has to do with >> PCI-Express latency: It takes about the same time to send a single byte >> as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes >> the >> limiting factor. >> > In DtoH2, it is the data movement from VecNorm. The size of data >> is 8B, >> > which is just the sizeof(PetscScalar). >> This is most likely the result required for the control flow on the CPU. >> > In DtoD1, it is the data movement from VecAXPY. The size of data is >> > 17.952MB, which is exactly sizeof(PetscScalar)*length(b). >> This is a vector assignment. If I remember correctly, it uses the >> memcpy-routines and hence shows up as a separate DtoD instead of just a >> kernel. It matches the time required for scal_kernel_val (scaling a >> vector by a scalar), so it runs at full bandwidth on the GPU. >> > However, I do not understand the number in HostToDevice in gmres for >> > np=1. The size of data movement is 1.032KB. I thought this is >> related to >> > the updated upper Hessenberg matrix, but the number does not >> match. Can >> > anyone help me understand the data movement of HToD in GMRES for >> np=1? >> 1032 = (128+1)*8, so this might be some auxiliary work information on >> the GPU. I could figure out the exact source of these transfers, but >> that is some effort. Let me know whether this is important information >> for you, then I can do it. >> Best regards, >> Karli >> > >> > Thank you. >> > >> > Best, >> > Xiangdong >> > >> > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp > >> > >> wrote: >> > >> > Hi, >> > >> > as you can see from the screenshot, the communication is >> merely for >> > scalars from the dot-products and/or norms. These are needed >> on the >> > host >> > for the control flow and convergence checks and is true for any >> > iterative solver. >> > >> > Best regards, >> > Karli >> > >> > >> > >> > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: >> > > >> > > >> > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. >> > >> > >> > > >> >>> wrote: >> > > >> > > >> > > 1) What preconditioner are you using? If any. >> > > >> > > Currently I am using none as I want to understand how >> gmres works >> > on GPU. >> > > >> > > >> > > 2) Where/how are you getting this information >> about the >> > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like >> to utilize >> > > this same sort of information to plan future >> optimizations. >> > > >> > > I am using nvprof and nvvp from cuda toolkit. It looks >> like there >> > are >> > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per >> iteration for np=1 >> > > case. See the attached snapshots. >> > > >> > > 3) Are you using more than 1 MPI rank? >> > > >> > > >> > > I tried both np=1 and np=2. Attached please find snapshots >> from >> > nvvp for >> > > both np=1 and np=2 cases. The figures showing gpu calls >> with two >> > pure >> > > gmres iterations. >> > > >> > > Thanks. >> > > Xiangdong >> > > >> > > >> > > If you use the master branch (which we highly >> recommend for >> > > anyone using GPUs and PETSc) the -log_view option will log >> > > communication between CPU and GPU and display it in >> the summary >> > > table. This is useful for seeing exactly what >> operations are >> > doing >> > > vector communication between the CPU/GPU. >> > > >> > > We welcome all feedback on the GPUs since it previously >> > has only >> > > been lightly used. >> > > >> > > Barry >> > > >> > > >> > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via petsc-users >> > > > > > >> > > > >>> >> > wrote: >> > > > >> > > > Hello everyone, >> > > > >> > > > I am new to petsc gpu and have a simple question. >> > > > >> > > > When I tried to solve Ax=b where A is >> MATAIJCUSPARSE and b >> > and x >> > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found >> that >> > during >> > > each krylov iteration, there are one call MemCpy(HtoD) and >> > one call >> > > MemCpy(DtoH). Does that mean the Krylov solve is not >> 100% on >> > GPU and >> > > the solve still needs some work from CPU? What are these >> > MemCpys for >> > > during the each iteration? >> > > > >> > > > Thank you. >> > > > >> > > > Best, >> > > > Xiangdong >> > > >> > From epscodes at gmail.com Tue Jul 23 21:01:41 2019 From: epscodes at gmail.com (Xiangdong) Date: Tue, 23 Jul 2019 22:01:41 -0400 Subject: [petsc-users] MemCpy (HtoD and DtoH) in Krylov solver In-Reply-To: <793A6A41-FAC1-4E61-9905-539B18BEE392@mcs.anl.gov> References: <159E2FDD-4439-4051-8CBC-024E3056F831@anl.gov> <793A6A41-FAC1-4E61-9905-539B18BEE392@mcs.anl.gov> Message-ID: Thanks, Barry and Karli! You comments are very helpful. At this point, I just started learning how to load the linear solve to GPU. Leave a copy of matrix on CPU is fine at the moment for me. If the GPUs I am going to work on are Nvidia GPUs, can AIJVIENNACL and AIJCUSPARSE provide very similar performance based on your experience? Thank you. Xiangdong On Tue, Jul 23, 2019 at 6:50 PM Smith, Barry F. wrote: > > > > On Jul 23, 2019, at 12:24 PM, Karl Rupp wrote: > > > > Hi, > > > >> I have two quick questions related to run gpu solvers. > >> 1) # of MPI processes vs # of GPUs. Is it true that we should set these > two numbers equal if most of computations are done on GPU? For one case I > tested, with only one GPU, running with np=2 is 15% slower than np=1 > (probably due to mpi communications). I am curious in what case, one will > benefit by having more mpi processes than number of GPUs. > > > > yes, usually you want to have the same number of MPI processes as the > number of GPUs on each node, unless you have a lot of work for the CPU and > only little work for the GPU. If you oversubscribe the GPU, it will not get > any faster (unlike multi-core CPUs); instead, the extra management overhead > will make it slow down. > > > > > >> 2) When I use MatLoad(A,viewer) to load a binary format data into a > aijcusparse format A, how many matrices are created? Does it involve > creating a intermediate aij matrix A' on CPU then convert it to aijcuspare > A on GPU? I am not sure whether such intermediate aij matrix exist. If yes, > What is the life time of this matrix? Is it destroyed right after the > converting? > > > > GPU matrices are essentially AIJ matrices with additional GPU data > members. That is, if you MatLoad(), the data will be copied into the CPU > buffers first and then pushed down to the GPU when needed. The CPU buffers > will never be free'd, but might be updated from the latest GPU data to > allow for fallback operations that have no GPU implementation. > > Eventually we should provide a mechanism that allows users to free the > CPU side of matrices (and vectors) if the user knows that they will never > need them again on the CPU. Honestly we are still working on the basics, > not yet at this level of optimization. If you would benefit from such a > feature now feel free to make a pull request. > > Barry > > > > > > > Best regards, > > Karli > > > > > > > > > > > >> On Fri, Jul 19, 2019 at 12:08 PM Karl Rupp > wrote: > >> Hi Xiangdong, > >> > I can understand some of the numbers, but not the HtoD case. > >> > > >> > In DtoH1, it is the data movement from VecMDot. The size of data > is > >> > 8.192KB, which is sizeof(PetscScalar) * MDOT_WORKGROUP_NUM * 8 = > >> 8*128*8 > >> > = 8192. My question is: instead of calling cublasDdot nv times, > >> why do > >> > you implement your own kernels? I guess it must be for > >> performance, but > >> > can you explain a little more? > >> Yes, this is a performance optimization. We've used several > >> dot-products > >> (suffers from kernel launch latency) as well as > matrix-vector-products > >> (suffers extra matrix setup) in the past; in both cases, there was > >> extra > >> memory traffic, thus impacting performance. > >> The reason why the data size is 8192 is to get around a separate > >> reduction stage on the GPU (i.e. a second kernel launch). By moving > the > >> data to the CPU and doing the reduction there, one is faster than > doing > >> it on the GPU and then moving only a few numbers. This has to do with > >> PCI-Express latency: It takes about the same time to send a single > byte > >> as sending a few kilobytes. Only beyond ~10 KB the bandwidth becomes > >> the > >> limiting factor. > >> > In DtoH2, it is the data movement from VecNorm. The size of data > >> is 8B, > >> > which is just the sizeof(PetscScalar). > >> This is most likely the result required for the control flow on the > CPU. > >> > In DtoD1, it is the data movement from VecAXPY. The size of data > is > >> > 17.952MB, which is exactly sizeof(PetscScalar)*length(b). > >> This is a vector assignment. If I remember correctly, it uses the > >> memcpy-routines and hence shows up as a separate DtoD instead of > just a > >> kernel. It matches the time required for scal_kernel_val (scaling a > >> vector by a scalar), so it runs at full bandwidth on the GPU. > >> > However, I do not understand the number in HostToDevice in gmres > for > >> > np=1. The size of data movement is 1.032KB. I thought this is > >> related to > >> > the updated upper Hessenberg matrix, but the number does not > >> match. Can > >> > anyone help me understand the data movement of HToD in GMRES for > >> np=1? > >> 1032 = (128+1)*8, so this might be some auxiliary work information on > >> the GPU. I could figure out the exact source of these transfers, but > >> that is some effort. Let me know whether this is important > information > >> for you, then I can do it. > >> Best regards, > >> Karli > >> > > >> > Thank you. > >> > > >> > Best, > >> > Xiangdong > >> > > >> > On Thu, Jul 18, 2019 at 1:14 PM Karl Rupp >> > >> > >> > wrote: > >> > > >> > Hi, > >> > > >> > as you can see from the screenshot, the communication is > >> merely for > >> > scalars from the dot-products and/or norms. These are needed > >> on the > >> > host > >> > for the control flow and convergence checks and is true for > any > >> > iterative solver. > >> > > >> > Best regards, > >> > Karli > >> > > >> > > >> > > >> > On 7/18/19 3:11 PM, Xiangdong via petsc-users wrote: > >> > > > >> > > > >> > > On Thu, Jul 18, 2019 at 5:11 AM Smith, Barry F. > >> > > >> > > >> > > > >> >>> wrote: > >> > > > >> > > > >> > > 1) What preconditioner are you using? If any. > >> > > > >> > > Currently I am using none as I want to understand how > >> gmres works > >> > on GPU. > >> > > > >> > > > >> > > 2) Where/how are you getting this information > >> about the > >> > > MemCpy(HtoD) and one call MemCpy(DtoH)? We might like > >> to utilize > >> > > this same sort of information to plan future > >> optimizations. > >> > > > >> > > I am using nvprof and nvvp from cuda toolkit. It looks > >> like there > >> > are > >> > > one MemCpy(HtoD) and three MemCpy(DtoH) calls per > >> iteration for np=1 > >> > > case. See the attached snapshots. > >> > > > >> > > 3) Are you using more than 1 MPI rank? > >> > > > >> > > > >> > > I tried both np=1 and np=2. Attached please find snapshots > >> from > >> > nvvp for > >> > > both np=1 and np=2 cases. The figures showing gpu calls > >> with two > >> > pure > >> > > gmres iterations. > >> > > > >> > > Thanks. > >> > > Xiangdong > >> > > > >> > > > >> > > If you use the master branch (which we highly > >> recommend for > >> > > anyone using GPUs and PETSc) the -log_view option will > log > >> > > communication between CPU and GPU and display it in > >> the summary > >> > > table. This is useful for seeing exactly what > >> operations are > >> > doing > >> > > vector communication between the CPU/GPU. > >> > > > >> > > We welcome all feedback on the GPUs since it > previously > >> > has only > >> > > been lightly used. > >> > > > >> > > Barry > >> > > > >> > > > >> > > > On Jul 16, 2019, at 9:05 PM, Xiangdong via > petsc-users > >> > > >> >> > > >> > >> >> >>> > >> > wrote: > >> > > > > >> > > > Hello everyone, > >> > > > > >> > > > I am new to petsc gpu and have a simple question. > >> > > > > >> > > > When I tried to solve Ax=b where A is > >> MATAIJCUSPARSE and b > >> > and x > >> > > are VECSEQCUDA with GMRES(or GCR) and pcnone, I found > >> that > >> > during > >> > > each krylov iteration, there are one call MemCpy(HtoD) > and > >> > one call > >> > > MemCpy(DtoH). Does that mean the Krylov solve is not > >> 100% on > >> > GPU and > >> > > the solve still needs some work from CPU? What are > these > >> > MemCpys for > >> > > during the each iteration? > >> > > > > >> > > > Thank you. > >> > > > > >> > > > Best, > >> > > > Xiangdong > >> > > > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.appel17 at imperial.ac.uk Thu Jul 25 11:29:41 2019 From: t.appel17 at imperial.ac.uk (Thibaut Appel) Date: Thu, 25 Jul 2019 17:29:41 +0100 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> Message-ID: <45999109-e951-50c8-ed25-8814838ba58d@imperial.ac.uk> Hi Hong, That sounds like a more reasonable approach, I had no idea the PETSc/MUMPS interface could provide such a level of control on the solve process. Therefore, after assembling the matrix A_0, I should do something along the lines of: MatGetFactor(A_0, MATSOLVERMUMPS, MAT_FACTOR_LU, F) MatLUFactorSymbolic(F, A_0, NULL, NULL, info) MatLUFactorNumeric(F, A_0, info) and then call MatSolve? However I don't understand, I thought F would remain the same during the whole process but it's an input parameter of MatSolve so I'd need one F_m for each A_m? Which is not what you mentioned (do one symbolic factorization only) On a side note, after preallocating and assembling the first matrix, should I create/assemble all the others with MatDuplicate(A_0, MAT_DO_NOT_COPY_VALUES, A_m) Calls to MatSetValues( ... ) MatAssemblyBegin(A_m, MAT_FINAL_ASSEMBLY) MatAssemblyEnd(A_m, MAT_FINAL_ASSEMBLY) Is that the recommended/most scalable way of duplicating a matrix + its non-zero structure? Thank you for your support and suggestions, Thibaut On 23/07/2019 18:38, Zhang, Hong wrote: > Thibaut: > > Thanks for taking the time. I would typically run that on a small > cluster node of 16 or 32 physical cores with 2 or 4 sockets. I use > 16 or > 32 MPI ranks and bind them to cores. > > The matrices would ALL have the same size and the same nonzero > structure > - it's just a few numerical values that would differ. > > You may do one symbolic factorization of A_m, use it in the m-i loop: > - numeric factorization of A_m > - solve A_m x_m,i = b_m,i > in mumps, numeric factorization and solve are scalable. Repeated > numeric factorization of A_m are likely faster than reading data files > from the disc. > Hong > > > This is a good point you've raised as I don't think MUMPS is able to > exploit that - I asked the question in their users list just to be > sure. > There are some options in SuperLU dist to reuse permutation > arrays, but > there's no I/O for that solver. And the native PETSc LU solver is not > parallel? > > I'm using high-order finite differences so I'm suffering from a > lot of > fill-in, one of the reasons why storing factorizations in RAM is not > viable. In comparison, I have almost unlimited disk space. > > I'm aware my need might seem counter-intuitive, but I'm really > willing > to sacrifice performance in the I/O part. My code is already heavily > based on PETSc (preallocation, assembly for matrices/vectors) coupled > with MUMPS I'm minimizing the loss of efficiency. > > Thibaut > > On 23/07/2019 17:13, Smith, Barry F. wrote: > >? ? What types of computing systems will you be doing the > computations? Roughly how many MPI_ranks? > > > > Are the matrices all the same size? Do they have the same or > different nonzero structures? Would it be possible to use the same > symbolic representation for all of them and just have different > numerical values? > > > >? ? Clusters and large scale computing centers are notoriously > terrible at IO; often IO is orders of magnitude slower than > compute/memory making this type of workflow unrealistically slow. > From a cost analysis point of view often just buying lots of > memory might be the most efficacious approach. > > > >? ? That said, what you suggest might require only a few lines of > code (though determining where to put them is the tricky part) > depending on the MUMPS interface for saving a filer to disk. What > we would do is keep the PETSc wrapper that lives around the MUMPS > matrix Mat_MUMPS but using the MUMPS API save the information in > the DMUMPS_STRUC_C id; and then reload it when needed. > > > >? ? The user level API could be something like > > > >? ? MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they > would just money with DMUMPS_STRUC_C id; item. > > > > > >? ? Barry > > > > > >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users > > wrote: > >> > >> Dear PETSc users, > >> > >> I need to solve several linear systems successively, with LU > factorization, as part of an iterative process in my Fortran > application code. > >> > >> The process would solve M systems (A_m)(x_m,i) = (b_m,i) for > m=1,M at each iteration i, but computing the LU factorization of > A_m only once. > >> The RHSs (b_m,i+1) are computed from all the different (x_m,i) > and all depend upon each other. > >> > >> The way I envisage to perform that is to use MUMPS to compute, > successively, each of the LU factorizations (m) in parallel and > store the factors on disk, creating/assembling/destroying the > matrices A_m on the go. > >> Then whenever needed, read the factors in parallel to solve the > systems. Since version 5.2, MUMPS has a save/restore feature that > allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf > p.20, 24 and 58. > >> > >> In its current state, the PETSc/MUMPS interface does not > incorporate that feature. I'm an advanced Fortran programmer but > not in C so I don't think I would do an amazing job having a go > inside src/mat/impls/aij/mpi/mumps/mumps.c. > >> > >> I was picturing something like creating as many KSP objects as > linear systems to be solved, with some sort of flag to force the > storage of LU factors on disk after the first call to KSPSolve. > Then keep calling KSPSolve as many times as needed. > >> > >> Would you support such a feature? > >> > >> Thanks for your support, > >> > >> Thibaut > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jul 25 13:30:35 2019 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Thu, 25 Jul 2019 18:30:35 +0000 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: <45999109-e951-50c8-ed25-8814838ba58d@imperial.ac.uk> References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> <45999109-e951-50c8-ed25-8814838ba58d@imperial.ac.uk> Message-ID: Thibaut: I'm writing a simple example using KSP directly -- will send to you soon. Hong Hi Hong, That sounds like a more reasonable approach, I had no idea the PETSc/MUMPS interface could provide such a level of control on the solve process. Therefore, after assembling the matrix A_0, I should do something along the lines of: MatGetFactor(A_0, MATSOLVERMUMPS, MAT_FACTOR_LU, F) MatLUFactorSymbolic(F, A_0, NULL, NULL, info) MatLUFactorNumeric(F, A_0, info) and then call MatSolve? However I don't understand, I thought F would remain the same during the whole process but it's an input parameter of MatSolve so I'd need one F_m for each A_m? Which is not what you mentioned (do one symbolic factorization only) On a side note, after preallocating and assembling the first matrix, should I create/assemble all the others with MatDuplicate(A_0, MAT_DO_NOT_COPY_VALUES, A_m) Calls to MatSetValues( ... ) MatAssemblyBegin(A_m, MAT_FINAL_ASSEMBLY) MatAssemblyEnd(A_m, MAT_FINAL_ASSEMBLY) Is that the recommended/most scalable way of duplicating a matrix + its non-zero structure? Thank you for your support and suggestions, Thibaut On 23/07/2019 18:38, Zhang, Hong wrote: Thibaut: Thanks for taking the time. I would typically run that on a small cluster node of 16 or 32 physical cores with 2 or 4 sockets. I use 16 or 32 MPI ranks and bind them to cores. The matrices would ALL have the same size and the same nonzero structure - it's just a few numerical values that would differ. You may do one symbolic factorization of A_m, use it in the m-i loop: - numeric factorization of A_m - solve A_m x_m,i = b_m,i in mumps, numeric factorization and solve are scalable. Repeated numeric factorization of A_m are likely faster than reading data files from the disc. Hong This is a good point you've raised as I don't think MUMPS is able to exploit that - I asked the question in their users list just to be sure. There are some options in SuperLU dist to reuse permutation arrays, but there's no I/O for that solver. And the native PETSc LU solver is not parallel? I'm using high-order finite differences so I'm suffering from a lot of fill-in, one of the reasons why storing factorizations in RAM is not viable. In comparison, I have almost unlimited disk space. I'm aware my need might seem counter-intuitive, but I'm really willing to sacrifice performance in the I/O part. My code is already heavily based on PETSc (preallocation, assembly for matrices/vectors) coupled with MUMPS I'm minimizing the loss of efficiency. Thibaut On 23/07/2019 17:13, Smith, Barry F. wrote: > What types of computing systems will you be doing the computations? Roughly how many MPI_ranks? > > Are the matrices all the same size? Do they have the same or different nonzero structures? Would it be possible to use the same symbolic representation for all of them and just have different numerical values? > > Clusters and large scale computing centers are notoriously terrible at IO; often IO is orders of magnitude slower than compute/memory making this type of workflow unrealistically slow. From a cost analysis point of view often just buying lots of memory might be the most efficacious approach. > > That said, what you suggest might require only a few lines of code (though determining where to put them is the tricky part) depending on the MUMPS interface for saving a filer to disk. What we would do is keep the PETSc wrapper that lives around the MUMPS matrix Mat_MUMPS but using the MUMPS API save the information in the DMUMPS_STRUC_C id; and then reload it when needed. > > The user level API could be something like > > MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they would just money with DMUMPS_STRUC_C id; item. > > > Barry > > >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users > wrote: >> >> Dear PETSc users, >> >> I need to solve several linear systems successively, with LU factorization, as part of an iterative process in my Fortran application code. >> >> The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at each iteration i, but computing the LU factorization of A_m only once. >> The RHSs (b_m,i+1) are computed from all the different (x_m,i) and all depend upon each other. >> >> The way I envisage to perform that is to use MUMPS to compute, successively, each of the LU factorizations (m) in parallel and store the factors on disk, creating/assembling/destroying the matrices A_m on the go. >> Then whenever needed, read the factors in parallel to solve the systems. Since version 5.2, MUMPS has a save/restore feature that allows that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and 58. >> >> In its current state, the PETSc/MUMPS interface does not incorporate that feature. I'm an advanced Fortran programmer but not in C so I don't think I would do an amazing job having a go inside src/mat/impls/aij/mpi/mumps/mumps.c. >> >> I was picturing something like creating as many KSP objects as linear systems to be solved, with some sort of flag to force the storage of LU factors on disk after the first call to KSPSolve. Then keep calling KSPSolve as many times as needed. >> >> Would you support such a feature? >> >> Thanks for your support, >> >> Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: From hong at aspiritech.org Thu Jul 25 15:25:42 2019 From: hong at aspiritech.org (hong at aspiritech.org) Date: Thu, 25 Jul 2019 15:25:42 -0500 Subject: [petsc-users] Solving a sequence of linear systems stored on disk with MUMPS In-Reply-To: References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> <45999109-e951-50c8-ed25-8814838ba58d@imperial.ac.uk> Message-ID: Thibaut: I added an example (in the branch hzhang/add-ksp-tutorials-ex6/master) https://bitbucket.org/petsc/petsc/commits/cf847786fd804b3606d0281d404c4763f36fe475?at=hzhang/add-ksp-tutorials-ex6/master You can run it with mpiexec -n 2 ./ex6 -num_numfac 2 -pc_type lu -pc_factor_mat_solver_type mumps -ksp_monitor -log_view ... MatLUFactorSym 1 1.0 3.5911e-03 MatLUFactorNum 2 1.0 6.3920e-03 This shows the code does one symbolic factorization and two numeric factorizations. For your convenience, the code ex6.c is attached below. Let me know if you have encounter any problems. Hong On Thu, Jul 25, 2019 at 1:30 PM Zhang, Hong via petsc-users < petsc-users at mcs.anl.gov> wrote: > Thibaut: > I'm writing a simple example using KSP directly -- will send to you soon. > Hong > >> Hi Hong, >> >> That sounds like a more reasonable approach, I had no idea the >> PETSc/MUMPS interface could provide such a level of control on the solve >> process. Therefore, after assembling the matrix A_0, I should do something >> along the lines of: >> >> MatGetFactor(A_0, MATSOLVERMUMPS, MAT_FACTOR_LU, F) >> >> MatLUFactorSymbolic(F, A_0, NULL, NULL, info) >> >> MatLUFactorNumeric(F, A_0, info) >> >> and then call MatSolve? However I don't understand, I thought F would >> remain the same during the whole process but it's an input parameter of >> MatSolve so I'd need one F_m for each A_m? Which is not what you mentioned >> (do one symbolic factorization only) >> >> >> On a side note, after preallocating and assembling the first matrix, >> should I create/assemble all the others with >> >> MatDuplicate(A_0, MAT_DO_NOT_COPY_VALUES, A_m) >> >> Calls to MatSetValues( ... ) >> MatAssemblyBegin(A_m, MAT_FINAL_ASSEMBLY) >> MatAssemblyEnd(A_m, MAT_FINAL_ASSEMBLY) >> >> Is that the recommended/most scalable way of duplicating a matrix + its >> non-zero structure? >> >> >> Thank you for your support and suggestions, >> >> Thibaut >> >> >> On 23/07/2019 18:38, Zhang, Hong wrote: >> >> Thibaut: >> >>> Thanks for taking the time. I would typically run that on a small >>> cluster node of 16 or 32 physical cores with 2 or 4 sockets. I use 16 or >>> 32 MPI ranks and bind them to cores. >>> >>> The matrices would ALL have the same size and the same nonzero structure >>> - it's just a few numerical values that would differ. >>> >> You may do one symbolic factorization of A_m, use it in the m-i loop: >> - numeric factorization of A_m >> - solve A_m x_m,i = b_m,i >> in mumps, numeric factorization and solve are scalable. Repeated numeric >> factorization of A_m are likely faster than reading data files from the >> disc. >> Hong >> >>> >>> This is a good point you've raised as I don't think MUMPS is able to >>> exploit that - I asked the question in their users list just to be sure. >>> There are some options in SuperLU dist to reuse permutation arrays, but >>> there's no I/O for that solver. And the native PETSc LU solver is not >>> parallel? >>> >>> I'm using high-order finite differences so I'm suffering from a lot of >>> fill-in, one of the reasons why storing factorizations in RAM is not >>> viable. In comparison, I have almost unlimited disk space. >>> >>> I'm aware my need might seem counter-intuitive, but I'm really willing >>> to sacrifice performance in the I/O part. My code is already heavily >>> based on PETSc (preallocation, assembly for matrices/vectors) coupled >>> with MUMPS I'm minimizing the loss of efficiency. >>> >>> Thibaut >>> >>> On 23/07/2019 17:13, Smith, Barry F. wrote: >>> > What types of computing systems will you be doing the computations? >>> Roughly how many MPI_ranks? >>> > >>> > Are the matrices all the same size? Do they have the same or different >>> nonzero structures? Would it be possible to use the same symbolic >>> representation for all of them and just have different numerical values? >>> > >>> > Clusters and large scale computing centers are notoriously terrible >>> at IO; often IO is orders of magnitude slower than compute/memory making >>> this type of workflow unrealistically slow. From a cost analysis point of >>> view often just buying lots of memory might be the most efficacious >>> approach. >>> > >>> > That said, what you suggest might require only a few lines of code >>> (though determining where to put them is the tricky part) depending on the >>> MUMPS interface for saving a filer to disk. What we would do is keep the >>> PETSc wrapper that lives around the MUMPS matrix Mat_MUMPS but using the >>> MUMPS API save the information in the DMUMPS_STRUC_C id; and then reload it >>> when needed. >>> > >>> > The user level API could be something like >>> > >>> > MatMumpsSaveToDisk(Mat) and MatMumpsLoadFromDisk(Mat) they would >>> just money with DMUMPS_STRUC_C id; item. >>> > >>> > >>> > Barry >>> > >>> > >>> >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >> >>> >> Dear PETSc users, >>> >> >>> >> I need to solve several linear systems successively, with LU >>> factorization, as part of an iterative process in my Fortran application >>> code. >>> >> >>> >> The process would solve M systems (A_m)(x_m,i) = (b_m,i) for m=1,M at >>> each iteration i, but computing the LU factorization of A_m only once. >>> >> The RHSs (b_m,i+1) are computed from all the different (x_m,i) and >>> all depend upon each other. >>> >> >>> >> The way I envisage to perform that is to use MUMPS to compute, >>> successively, each of the LU factorizations (m) in parallel and store the >>> factors on disk, creating/assembling/destroying the matrices A_m on the go. >>> >> Then whenever needed, read the factors in parallel to solve the >>> systems. Since version 5.2, MUMPS has a save/restore feature that allows >>> that, see http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 and >>> 58. >>> >> >>> >> In its current state, the PETSc/MUMPS interface does not incorporate >>> that feature. I'm an advanced Fortran programmer but not in C so I don't >>> think I would do an amazing job having a go inside >>> src/mat/impls/aij/mpi/mumps/mumps.c. >>> >> >>> >> I was picturing something like creating as many KSP objects as linear >>> systems to be solved, with some sort of flag to force the storage of LU >>> factors on disk after the first call to KSPSolve. Then keep calling >>> KSPSolve as many times as needed. >>> >> >>> >> Would you support such a feature? >>> >> >>> >> Thanks for your support, >>> >> >>> >> Thibaut >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex6.c Type: application/octet-stream Size: 3320 bytes Desc: not available URL: From dmcvamos at udel.edu Fri Jul 26 14:55:08 2019 From: dmcvamos at udel.edu (Mingchang Ding) Date: Fri, 26 Jul 2019 13:55:08 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style Message-ID: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> Hi, all I am trying to apply LDG to discretizing 1D diffusion term u_{xx} with Dirichlet boundary condition. The difficulty I have is that I want to zero all entries (except the diagonal one needs to be set as 1.0) of the beginning and ending DOF rows of the sparse matrix. I have tried the MatZeroRows and MatZeroRowsStencil. However, I can only change the very first and last row of the matrix with them. This isn?t right when DOF > 1 because I need to zero the first DOF rows (the diagonal should be set as identity). I was wondering if there is anyway I can take to zero the first and last DOF rows (with identity matrix diagonal) in a block-wise style. Let me know if any further explanation is needed. Thank you in advance! Mingchang Ding Graduate Student EWG 111 Department of Mathematical Sciences University of Delaware -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 26 15:11:44 2019 From: jed at jedbrown.org (Jed Brown) Date: Fri, 26 Jul 2019 14:11:44 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style In-Reply-To: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> References: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> Message-ID: <87sgqsd08v.fsf@jedbrown.org> Can you give an example of what you are trying? These functions should be capable of handling any set of rows. Mingchang Ding via petsc-users writes: > Hi, all > > I am trying to apply LDG to discretizing 1D diffusion term u_{xx} with Dirichlet boundary condition. The difficulty I have is that I want to zero all entries (except the diagonal one needs to be set as 1.0) of the beginning and ending DOF rows of the sparse matrix. > > I have tried the MatZeroRows and MatZeroRowsStencil. However, I can only change the very first and last row of the matrix with them. This isn?t right when DOF > 1 because I need to zero the first DOF rows (the diagonal should be set as identity). > > I was wondering if there is anyway I can take to zero the first and last DOF rows (with identity matrix diagonal) in a block-wise style. > > Let me know if any further explanation is needed. > > Thank you in advance! > > Mingchang Ding > Graduate Student > EWG 111 > Department of Mathematical Sciences > University of Delaware From jed at jedbrown.org Fri Jul 26 16:21:21 2019 From: jed at jedbrown.org (Jed Brown) Date: Fri, 26 Jul 2019 15:21:21 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style In-Reply-To: <1F942EEC-BF92-466C-88A6-1BD4F9EC8420@udel.edu> References: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> <87sgqsd08v.fsf@jedbrown.org> <1F942EEC-BF92-466C-88A6-1BD4F9EC8420@udel.edu> Message-ID: <87imrocx0u.fsf@jedbrown.org> Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Mingchang Ding writes: > Thank you for your quick reply. The code I have now is attached. The sparse matrix obtained from LDG discretization to u_xx is assembled in SetupMat routine of setup.h. When I choose my polynomial degree nk =1 (DOF = 2), only the first and last row of the output sparse matrix is as what I want. Again, I want to impose essential (Dirichlet) boundary condition to the problem. > > Let me know if any explanation of the question is needed. It appears that you are calling MatZeroRows with rows = {0, info.mx-1}, which will only affect the first and last rows. The other entries are how you have assembled them (with the matrix-matrix product). PetscInt nrows = 0; PetscInt *rows; ierr = DMDAGetLocalInfo(da, &info);CHKERRQ(ierr); PetscMalloc1(info.mx, &rows); for (j=info.xs; jMass_inv, 1, &j, 1, &j, &inv[0][0], INSERT_VALUES);CHKERRQ(ierr); } [...] MatZeroRows(appctx->J_dif, nrows, rows, 1.0, NULL, NULL); From dmcvamos at udel.edu Fri Jul 26 16:32:28 2019 From: dmcvamos at udel.edu (Mingchang Ding) Date: Fri, 26 Jul 2019 15:32:28 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style In-Reply-To: <87imrocx0u.fsf@jedbrown.org> References: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> <87sgqsd08v.fsf@jedbrown.org> <1F942EEC-BF92-466C-88A6-1BD4F9EC8420@udel.edu> <87imrocx0u.fsf@jedbrown.org> Message-ID: > On Jul 26, 2019, at 3:21 PM, Jed Brown wrote: > > Please always use "reply-all" so that your messages go to the list. > This is standard mailing list etiquette. It is important to preserve > threading for people who find this discussion later and so that we do > not waste our time re-answering the same questions that have already > been answered in private side-conversations. You'll likely get an > answer faster that way too. Sorry for this. > Mingchang Ding writes: > >> Thank you for your quick reply. The code I have now is attached. The sparse matrix obtained from LDG discretization to u_xx is assembled in SetupMat routine of setup.h. When I choose my polynomial degree nk =1 (DOF = 2), only the first and last row of the output sparse matrix is as what I want. Again, I want to impose essential (Dirichlet) boundary condition to the problem. >> >> Let me know if any explanation of the question is needed. > It appears that you are calling MatZeroRows with rows = {0, info.mx-1}, > which will only affect the first and last rows. The other entries are > how you have assembled them (with the matrix-matrix product). Yes. I think so. Is there anyway I can fill in the rows array with all the first and last DOF rows index efficiently instead of only filling in one row index at one time? > PetscInt nrows = 0; > PetscInt *rows; > > ierr = DMDAGetLocalInfo(da, &info);CHKERRQ(ierr); > > PetscMalloc1(info.mx, &rows); > > for (j=info.xs; j > if (j == 0) { > rows[nrows++] = j; > > [...] > > } > else if (j == info.mx - 1) { > rows[nrows++] = j; > > [...] > > > } > else { > > [...] > > > } > > /* Mass_inv */ > ierr = MatSetValuesBlocked(appctx->Mass_inv, 1, &j, 1, &j, &inv[0][0], INSERT_VALUES);CHKERRQ(ierr); > > } > > > [...] > > > MatZeroRows(appctx->J_dif, nrows, rows, 1.0, NULL, NULL); From dmcvamos at udel.edu Fri Jul 26 16:43:16 2019 From: dmcvamos at udel.edu (Mingchang Ding) Date: Fri, 26 Jul 2019 15:43:16 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style In-Reply-To: References: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> <87sgqsd08v.fsf@jedbrown.org> <1F942EEC-BF92-466C-88A6-1BD4F9EC8420@udel.edu> <87imrocx0u.fsf@jedbrown.org> Message-ID: > On Jul 26, 2019, at 3:32 PM, Mingchang Ding wrote: > > >> On Jul 26, 2019, at 3:21 PM, Jed Brown wrote: >> >> Please always use "reply-all" so that your messages go to the list. >> This is standard mailing list etiquette. It is important to preserve >> threading for people who find this discussion later and so that we do >> not waste our time re-answering the same questions that have already >> been answered in private side-conversations. You'll likely get an >> answer faster that way too. > > Sorry for this. > >> Mingchang Ding writes: >> >>> Thank you for your quick reply. The code I have now is attached. The sparse matrix obtained from LDG discretization to u_xx is assembled in SetupMat routine of setup.h. When I choose my polynomial degree nk =1 (DOF = 2), only the first and last row of the output sparse matrix is as what I want. Again, I want to impose essential (Dirichlet) boundary condition to the problem. >>> >>> Let me know if any explanation of the question is needed. >> It appears that you are calling MatZeroRows with rows = {0, info.mx-1}, >> which will only affect the first and last rows. The other entries are >> how you have assembled them (with the matrix-matrix product). > > Yes. I think so. Is there anyway I can fill in the rows array with all the first and last DOF rows index efficiently instead of only filling in one row index at one time? I think I misunderstood just now and know how to fix the problem. Thank you! >> PetscInt nrows = 0; >> PetscInt *rows; >> >> ierr = DMDAGetLocalInfo(da, &info);CHKERRQ(ierr); >> >> PetscMalloc1(info.mx, &rows); >> >> for (j=info.xs; j> >> if (j == 0) { >> rows[nrows++] = j; >> >> [...] >> >> } >> else if (j == info.mx - 1) { >> rows[nrows++] = j; >> >> [...] >> >> >> } >> else { >> >> [...] >> >> >> } >> >> /* Mass_inv */ >> ierr = MatSetValuesBlocked(appctx->Mass_inv, 1, &j, 1, &j, &inv[0][0], INSERT_VALUES);CHKERRQ(ierr); >> >> } >> >> >> [...] >> >> >> MatZeroRows(appctx->J_dif, nrows, rows, 1.0, NULL, NULL); > From jed at jedbrown.org Fri Jul 26 16:45:00 2019 From: jed at jedbrown.org (Jed Brown) Date: Fri, 26 Jul 2019 15:45:00 -0600 Subject: [petsc-users] Zero entries of a sparse matrix in a block-wise style In-Reply-To: References: <69667AF5-4D29-4834-B8A3-3333109E62E4@udel.edu> <87sgqsd08v.fsf@jedbrown.org> <1F942EEC-BF92-466C-88A6-1BD4F9EC8420@udel.edu> <87imrocx0u.fsf@jedbrown.org> Message-ID: <87d0hwcvxf.fsf@jedbrown.org> Mingchang Ding writes: >> It appears that you are calling MatZeroRows with rows = {0, info.mx-1}, >> which will only affect the first and last rows. The other entries are >> how you have assembled them (with the matrix-matrix product). > > Yes. I think so. Is there anyway I can fill in the rows array with all the first and last DOF rows index efficiently instead of only filling in one row index at one time? for (dof=0; dof References: <58C382E5-26BB-4960-83A9-F82CE372E6AD@anl.gov> <45999109-e951-50c8-ed25-8814838ba58d@imperial.ac.uk> Message-ID: <087f6c3b-9231-bb90-462c-7ceab317b4ac@imperial.ac.uk> Hi Hong, Thanks very much for that example I appreciate it. I'll test that in a few days and come back with questions if needed, Thibaut On 25/07/2019 21:25, hong at aspiritech.org wrote: > Thibaut: > I added an example (in the branch?hzhang/add-ksp-tutorials-ex6/master) > https://bitbucket.org/petsc/petsc/commits/cf847786fd804b3606d0281d404c4763f36fe475?at=hzhang/add-ksp-tutorials-ex6/master > > You can run it with > mpiexec -n 2 ./ex6 -num_numfac 2 -pc_type lu > -pc_factor_mat_solver_type mumps -ksp_monitor -log_view > ... > MatLUFactorSym ? ? ? ? 1 1.0 3.5911e-03 > MatLUFactorNum ? ? ? ? 2 1.0 6.3920e-03 > > This shows the code does one symbolic factorization and two numeric > factorizations. > For your convenience, the code ex6.c is attached below. > Let?me know if you have encounter any problems. > Hong > > > On Thu, Jul 25, 2019 at 1:30 PM Zhang, Hong via petsc-users > > wrote: > > Thibaut: > I'm writing a simple example using KSP directly -- will send to > you soon. > Hong > > Hi Hong, > > That sounds like a more reasonable approach, I had no idea the > PETSc/MUMPS interface could provide such a level of control on > the solve process. Therefore, after assembling the matrix A_0, > I should do something along the lines of: > > MatGetFactor(A_0, MATSOLVERMUMPS, MAT_FACTOR_LU, F) > > MatLUFactorSymbolic(F, A_0, NULL, NULL, info) > > MatLUFactorNumeric(F, A_0, info) > > and then call MatSolve? However I don't understand, I thought > F would remain the same during the whole process but it's an > input parameter of MatSolve so I'd need one F_m for each A_m? > Which is not what you mentioned (do one symbolic factorization > only) > > > On a side note, after preallocating and assembling the first > matrix, should I create/assemble all the others with > > MatDuplicate(A_0, MAT_DO_NOT_COPY_VALUES, A_m) > > Calls to MatSetValues( ... ) > > MatAssemblyBegin(A_m, MAT_FINAL_ASSEMBLY) > MatAssemblyEnd(A_m, MAT_FINAL_ASSEMBLY) > > Is that the recommended/most scalable way of duplicating a > matrix + its non-zero structure? > > > Thank you for your support and suggestions, > > Thibaut > > > On 23/07/2019 18:38, Zhang, Hong wrote: >> Thibaut: >> >> Thanks for taking the time. I would typically run that on >> a small >> cluster node of 16 or 32 physical cores with 2 or 4 >> sockets. I use 16 or >> 32 MPI ranks and bind them to cores. >> >> The matrices would ALL have the same size and the same >> nonzero structure >> - it's just a few numerical values that would differ. >> >> You may do one symbolic factorization of A_m, use it in the >> m-i loop: >> - numeric factorization of A_m >> - solve A_m x_m,i = b_m,i >> in mumps, numeric factorization and solve are scalable. >> Repeated numeric factorization of A_m are likely faster than >> reading data files from the disc. >> Hong >> >> >> This is a good point you've raised as I don't think MUMPS >> is able to >> exploit that - I asked the question in their users list >> just to be sure. >> There are some options in SuperLU dist to reuse >> permutation arrays, but >> there's no I/O for that solver. And the native PETSc LU >> solver is not >> parallel? >> >> I'm using high-order finite differences so I'm suffering >> from a lot of >> fill-in, one of the reasons why storing factorizations in >> RAM is not >> viable. In comparison, I have almost unlimited disk space. >> >> I'm aware my need might seem counter-intuitive, but I'm >> really willing >> to sacrifice performance in the I/O part. My code is >> already heavily >> based on PETSc (preallocation, assembly for >> matrices/vectors) coupled >> with MUMPS I'm minimizing the loss of efficiency. >> >> Thibaut >> >> On 23/07/2019 17:13, Smith, Barry F. wrote: >> >? ? What types of computing systems will you be doing >> the computations? Roughly how many MPI_ranks? >> > >> > Are the matrices all the same size? Do they have the >> same or different nonzero structures? Would it be >> possible to use the same symbolic representation for all >> of them and just have different numerical values? >> > >> >? ? Clusters and large scale computing centers are >> notoriously terrible at IO; often IO is orders of >> magnitude slower than compute/memory making this type of >> workflow unrealistically slow. From a cost analysis point >> of view often just buying lots of memory might be the >> most efficacious approach. >> > >> >? ? That said, what you suggest might require only a few >> lines of code (though determining where to put them is >> the tricky part) depending on the MUMPS interface for >> saving a filer to disk. What we would do is keep the >> PETSc wrapper that lives around the MUMPS matrix >> Mat_MUMPS but using the MUMPS API save the information in >> the DMUMPS_STRUC_C id; and then reload it when needed. >> > >> >? ? The user level API could be something like >> > >> >? ? MatMumpsSaveToDisk(Mat) and >> MatMumpsLoadFromDisk(Mat) they would just money with >> DMUMPS_STRUC_C id; item. >> > >> > >> >? ? Barry >> > >> > >> >> On Jul 23, 2019, at 9:24 AM, Thibaut Appel via >> petsc-users > > wrote: >> >> >> >> Dear PETSc users, >> >> >> >> I need to solve several linear systems successively, >> with LU factorization, as part of an iterative process in >> my Fortran application code. >> >> >> >> The process would solve M systems (A_m)(x_m,i) = >> (b_m,i) for m=1,M at each iteration i, but computing the >> LU factorization of A_m only once. >> >> The RHSs (b_m,i+1) are computed from all the different >> (x_m,i) and all depend upon each other. >> >> >> >> The way I envisage to perform that is to use MUMPS to >> compute, successively, each of the LU factorizations (m) >> in parallel and store the factors on disk, >> creating/assembling/destroying the matrices A_m on the go. >> >> Then whenever needed, read the factors in parallel to >> solve the systems. Since version 5.2, MUMPS has a >> save/restore feature that allows that, see >> http://mumps.enseeiht.fr/doc/userguide_5.2.1.pdf p.20, 24 >> and 58. >> >> >> >> In its current state, the PETSc/MUMPS interface does >> not incorporate that feature. I'm an advanced Fortran >> programmer but not in C so I don't think I would do an >> amazing job having a go inside >> src/mat/impls/aij/mpi/mumps/mumps.c. >> >> >> >> I was picturing something like creating as many KSP >> objects as linear systems to be solved, with some sort of >> flag to force the storage of LU factors on disk after the >> first call to KSPSolve. Then keep calling KSPSolve as >> many times as needed. >> >> >> >> Would you support such a feature? >> >> >> >> Thanks for your support, >> >> >> >> Thibaut >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jul 29 06:26:57 2019 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 29 Jul 2019 14:26:57 +0300 Subject: [petsc-users] Index set in Chaco Message-ID: Hello everyone, I am trying to partition a matrix by using chaco. By using ex15 from mat examples, I wrote the code below: MatPartitioningCreate(MPI_COMM_WORLD, &part); MatPartitioningSetAdjacency(part, A); MatPartitioningSetFromOptions(part); MatPartitioningApply(part, &is); ISView(is, PETSC_VIEWER_STDOUT_WORLD); ISDestroy(&is); MatPartitioningDestroy(&part); MatDestroy(&A); However, I always get zeros when I printed index set. I tried to change eigensolver but it doesn't work. Since this is the first time I am using Chaco, I couldn't understand where I did something wrong. Thanks, Eda -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Zhang-10 at tudelft.nl Mon Jul 29 11:02:23 2019 From: J.Zhang-10 at tudelft.nl (Jian Zhang - 3ME) Date: Mon, 29 Jul 2019 16:02:23 +0000 Subject: [petsc-users] Modifying dmplex dynamically Message-ID: Hi guys, I am trying to modify the data inside dmplex. For example, I would like to add one point (vertice, edge or cell) to the dmplex and also remove one point. Is it possible to do this in dmplex? If yes, which functions I should use? Thank you very much. Bests, Jian -------------- next part -------------- An HTML attachment was scrubbed... URL: From piotr.sierant at uj.edu.pl Mon Jul 29 12:23:09 2019 From: piotr.sierant at uj.edu.pl (Piotr Sierant) Date: Mon, 29 Jul 2019 17:23:09 +0000 Subject: [petsc-users] Full OpenMP strategy MUMPS Message-ID: <1564420989472.65477@uj.edu.pl> Hello everyone, I am trying to use PETSc with MUMPS in full openMP strategy motivated by data from SciPost Phys. 5, 045 (2018) which suggest that I can reduce RAM usage for particular system sizes in this way. I cannot get it to work in the way it is described at https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERMUMPS.html Having configured PETSc with the following options: Configure options --download-mpich --download-scalapack --download-cmake --with-openmp --download-metis --download-mumps --with-threadsafety --with-log=0 --with-debugging=0 --download-hwloc --with-blaslapack-dir=/opt/intel/Compiler/19.0/compilers_and_libraries_2019.4.243/linux/mkl (also with configured SLEPc which I need for shift-and-invert which requires LU decomposition from MUMPS) With OMP_NUM_THREADS=16 I execute: mpirun -n 2 ./my_program -mat_mumps_use_omp_threads 8 obtaining [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: number of OpenMP threads 8 can not be < 1 or > the MPI shared memory communicator size 1 (...) What should I do to be able to run the code with single (or few) MPI rank(s) and with MUMPS operating with 8 openMP threads? I will greatly appreciate any comments. Thanks, Piotr -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jul 29 12:58:23 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 29 Jul 2019 17:58:23 +0000 Subject: [petsc-users] Full OpenMP strategy MUMPS In-Reply-To: <1564420989472.65477@uj.edu.pl> References: <1564420989472.65477@uj.edu.pl> Message-ID: Please look at https://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSOLVERMUMPS.html it has a clearer explanation of the two ways one can use MUMPS and OpenMP ( note the petsc-dev in the URL). In your case you want to use OMP_NUM_THREADS=8 mpirun -n 2 ./my_program This will run two MPI processes of MUMPS each using 8 threads You cannot mix the OMP_NUM_THREADS= and the -mat_mumps_use_omp_threads option. They are for different situations. Barry > OMP_NUM_THREADS=16 I execute: > mpirun -n 2 ./my_program -mat_mumps_use_omp_threads 8 > > On Jul 29, 2019, at 12:23 PM, Piotr Sierant via petsc-users wrote: > > Hello everyone, > > I am trying to use PETSc with MUMPS in full openMP strategy motivated by data from > SciPost Phys. 5, 045 (2018) which suggest that I can reduce RAM usage for particular system sizes in this way. > I cannot get it to work in the way it is described at > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERMUMPS.html > Having configured PETSc with the following options: > Configure options --download-mpich --download-scalapack --download-cmake --with-openmp --download-metis --download-mumps --with-threadsafety --with-log=0 --with-debugging=0 --download-hwloc --with-blaslapack-dir=/opt/intel/Compiler/19.0/compilers_and_libraries_2019.4.243/linux/mkl > (also with configured SLEPc which I need for shift-and-invert which requires LU decomposition from MUMPS) With OMP_NUM_THREADS=16 I execute: > mpirun -n 2 ./my_program -mat_mumps_use_omp_threads 8 > obtaining > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: number of OpenMP threads 8 can not be < 1 or > the MPI shared memory communicator size 1 > (...) > What should I do to be able to run the code with single (or few) MPI rank(s) and with MUMPS operating with 8 openMP threads? I will greatly appreciate any comments. > > Thanks, Piotr > > > > > From bsmith at mcs.anl.gov Mon Jul 29 13:08:18 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 29 Jul 2019 18:08:18 +0000 Subject: [petsc-users] Index set in Chaco In-Reply-To: References: Message-ID: <2267CFB7-356B-4C8C-A86D-46BCBB16EEAB@anl.gov> What does MatPartitioningView() indicate? Are you using -mat_partitioning_type chaco ? > On Jul 29, 2019, at 6:26 AM, Eda Oktay via petsc-users wrote: > > Hello everyone, > > I am trying to partition a matrix by using chaco. By using ex15 from mat examples, I wrote the code below: > > MatPartitioningCreate(MPI_COMM_WORLD, &part); > MatPartitioningSetAdjacency(part, A); > MatPartitioningSetFromOptions(part); > MatPartitioningApply(part, &is); > ISView(is, PETSC_VIEWER_STDOUT_WORLD); > ISDestroy(&is); > MatPartitioningDestroy(&part); > MatDestroy(&A); > > However, I always get zeros when I printed index set. I tried to change eigensolver but it doesn't work. Since this is the first time I am using Chaco, I couldn't understand where I did something wrong. > > Thanks, > > Eda From knepley at gmail.com Mon Jul 29 14:39:01 2019 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Jul 2019 07:39:01 +1200 Subject: [petsc-users] Modifying dmplex dynamically In-Reply-To: References: Message-ID: On Tue, Jul 30, 2019 at 4:26 AM Jian Zhang - 3ME via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi guys, > > I am trying to modify the data inside dmplex. For example, I would like to > add one point (vertice, edge or cell) to the dmplex and also remove one > point. Is it possible to do this in dmplex? If yes, which functions I > should use? Thank you very much. > > These are data structures optimized for queries, not updates. It is possible to update them, for instance as we do in DMPlexInterpolate() or DMPlexAdapt(), but you basically recreate the structure. You also have to update auxiliary structures based on the topology. Sometimes the alteration can be really low overhead, like DMPlexCreateGhostCells(), but it depends on what is being done. What exactly do you want to do? Thanks, Matt > Bests, > > Jian > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Tue Jul 30 01:09:51 2019 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 30 Jul 2019 09:09:51 +0300 Subject: [petsc-users] Index set in Chaco In-Reply-To: <2267CFB7-356B-4C8C-A86D-46BCBB16EEAB@anl.gov> References: <2267CFB7-356B-4C8C-A86D-46BCBB16EEAB@anl.gov> Message-ID: Yes I use -mat_partitioning chaco. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 30 08:28:49 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 30 Jul 2019 13:28:49 +0000 Subject: [petsc-users] Index set in Chaco In-Reply-To: References: <2267CFB7-356B-4C8C-A86D-46BCBB16EEAB@anl.gov> Message-ID: <16B99C6C-05D9-4F51-A83F-10F17DD4C323@mcs.anl.gov> It requires -mat_partitioning_type chaco do you have the type in there? You may need to do some debugging, put a break point in MatPartitioningApply and use next and step to determine if it is actually calling MatPartitioningApply_Chaco Barry > On Jul 30, 2019, at 1:09 AM, Eda Oktay wrote: > > Yes I use -mat_partitioning chaco. From karl.linkui at gmail.com Tue Jul 30 10:23:53 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Tue, 30 Jul 2019 10:23:53 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Message-ID: Hi, Richard, We have a new question. Is there a limit for MatCreateMPIAIJ and MatSetValues? What I mean is that, we tried to create a sparse matrix and populate it with 50GB of data in one process, I got a crash and error saying object too big. Thank you for any insight. Regards, Karl On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi Kun and Karl, > > If you are using the AIJMKL matrix types and have a recent version of MKL, > the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which > are described at > > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > > The inspector-executor analysis routines take the AIJ (compressed sparse > row) format data from PETSc and then create a copy in an optimized, > internal layout used by MKL. We have to keep PETSc's own, AIJ > representation around, as it is needed for several operations that MKL does > not provide. This does, unfortunately, mean that roughly double (or more, > depending on what MKL decides to do) the amount of memory is required. The > reason you see the memory usage increase right when a MatMult() or > MatMultTranspose() operation occurs is that the we default to a "lazy" > approach to calling the analysis routine (mkl_sparse_optimize()) until an > operation that uses an MKL-provided kernel is requested. (You can use an > "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() > by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > > If memory is at enough of a premium for you that you can't afford the > extra copy used by the MKL inspector-executor routines, then I suggest > using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well > optimized for many cases (and even has some hand-optimized kernels using > Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should > try both AIJ and AIJMKL, anyway, to see which is faster for your > combination of problem and computing platform. > > Best regards, > Richard > > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: > > We also found that if we use MatCreateSeqAIJ, then no more memory increase > with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the > behavior is consistent. > > On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: > >> MatCreateMPIAIJMKL >> >> parallel and sequential exhibit the same behavior. In fact, we found that >> doing matmult will increase the memory by the size of matrix as well. >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >> >>> Karl: >>> What matrix format do you use? Run it in parallel or sequential? >>> Hong >>> >>> We used /proc/self/stat to track the resident set size during program >>>> run, and we saw the resident set size jumped by the size of the matrix >>>> right after we did matmulttranspose. >>>> >>>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> >>>>> Kun: >>>>> How do you know 'MatMultTranpose creates an extra memory copy of >>>>> matrix'? >>>>> Hong >>>>> >>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>>>> >>>>>> >>>>>> >>>>>> However we found out, MatMultTranpose create an extra memory copy of >>>>>> matrix for its operation. This extra memory copy is not stated everywhere >>>>>> in petsc manual. >>>>>> >>>>>> >>>>>> >>>>>> This basically double my memory requirement to solve my system. >>>>>> >>>>>> >>>>>> >>>>>> I remember mkl?s routine can do inplace matrix transpose vector >>>>>> product, without transposing the matrix itself. >>>>>> >>>>>> >>>>>> >>>>>> Is this always the case? Or there is way to make petsc to do inplace >>>>>> matrix transpose vector product. >>>>>> >>>>>> >>>>>> >>>>>> Any help is greatly appreciated. >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Kun >>>>>> >>>>>> >>>>>> >>>>>> Schlumberger-Private >>>>>> >>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 30 10:34:37 2019 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 31 Jul 2019 03:34:37 +1200 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Message-ID: On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi, Richard, > > We have a new question. Is there a limit for MatCreateMPIAIJ and > MatSetValues? What I mean is that, we tried to create a sparse matrix and > populate it with 50GB of data in one process, I got a crash and error > saying object too big. Thank you for any insight. > 1) Always send the complete error. 2) It sounds like you got an out of memory error for that process. Matt > Regards, > > Karl > > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi Kun and Karl, >> >> If you are using the AIJMKL matrix types and have a recent version of >> MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, >> which are described at >> >> >> https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines >> >> The inspector-executor analysis routines take the AIJ (compressed sparse >> row) format data from PETSc and then create a copy in an optimized, >> internal layout used by MKL. We have to keep PETSc's own, AIJ >> representation around, as it is needed for several operations that MKL does >> not provide. This does, unfortunately, mean that roughly double (or more, >> depending on what MKL decides to do) the amount of memory is required. The >> reason you see the memory usage increase right when a MatMult() or >> MatMultTranspose() operation occurs is that the we default to a "lazy" >> approach to calling the analysis routine (mkl_sparse_optimize()) until an >> operation that uses an MKL-provided kernel is requested. (You can use an >> "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() >> by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) >> >> If memory is at enough of a premium for you that you can't afford the >> extra copy used by the MKL inspector-executor routines, then I suggest >> using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well >> optimized for many cases (and even has some hand-optimized kernels using >> Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should >> try both AIJ and AIJMKL, anyway, to see which is faster for your >> combination of problem and computing platform. >> >> Best regards, >> Richard >> >> On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >> >> We also found that if we use MatCreateSeqAIJ, then no more memory >> increase with matrix vector multiplication. However, with >> MatCreateMPIAIJMKL, the behavior is consistent. >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: >> >>> MatCreateMPIAIJMKL >>> >>> parallel and sequential exhibit the same behavior. In fact, we found >>> that doing matmult will increase the memory by the size of matrix as well. >>> >>> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >>> >>>> Karl: >>>> What matrix format do you use? Run it in parallel or sequential? >>>> Hong >>>> >>>> We used /proc/self/stat to track the resident set size during program >>>>> run, and we saw the resident set size jumped by the size of the matrix >>>>> right after we did matmulttranspose. >>>>> >>>>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >>>>> petsc-users at mcs.anl.gov> wrote: >>>>> >>>>>> Kun: >>>>>> How do you know 'MatMultTranpose creates an extra memory copy of >>>>>> matrix'? >>>>>> Hong >>>>>> >>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> However we found out, MatMultTranpose create an extra memory copy of >>>>>>> matrix for its operation. This extra memory copy is not stated everywhere >>>>>>> in petsc manual. >>>>>>> >>>>>>> >>>>>>> >>>>>>> This basically double my memory requirement to solve my system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I remember mkl?s routine can do inplace matrix transpose vector >>>>>>> product, without transposing the matrix itself. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Is this always the case? Or there is way to make petsc to do inplace >>>>>>> matrix transpose vector product. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any help is greatly appreciated. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Kun >>>>>>> >>>>>>> >>>>>>> >>>>>>> Schlumberger-Private >>>>>>> >>>>>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Tue Jul 30 17:27:23 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Tue, 30 Jul 2019 17:27:23 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Message-ID: number of rows is 26326575. maximum column index is 36416250. number of nonzero coefficients is 6653750976, which amounts to 49.7GB for coefficients in PetscScalar and column index in PetscInt. I can run the program in 4 processes with this input but not single process. Here are the snap shots of the error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [0]PETSC ERROR: Memory requested 18446744058325389312 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: #2 PetscMallocA() line 390 in /petsc-3.10.4/src/sys/memory/mal.c [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #6 *** (my code) [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley wrote: > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hi, Richard, >> >> We have a new question. Is there a limit for MatCreateMPIAIJ and >> MatSetValues? What I mean is that, we tried to create a sparse matrix and >> populate it with 50GB of data in one process, I got a crash and error >> saying object too big. Thank you for any insight. >> > > 1) Always send the complete error. > > 2) It sounds like you got an out of memory error for that process. > > Matt > > >> Regards, >> >> Karl >> >> On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >>> Hi Kun and Karl, >>> >>> If you are using the AIJMKL matrix types and have a recent version of >>> MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, >>> which are described at >>> >>> >>> https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines >>> >>> The inspector-executor analysis routines take the AIJ (compressed sparse >>> row) format data from PETSc and then create a copy in an optimized, >>> internal layout used by MKL. We have to keep PETSc's own, AIJ >>> representation around, as it is needed for several operations that MKL does >>> not provide. This does, unfortunately, mean that roughly double (or more, >>> depending on what MKL decides to do) the amount of memory is required. The >>> reason you see the memory usage increase right when a MatMult() or >>> MatMultTranspose() operation occurs is that the we default to a "lazy" >>> approach to calling the analysis routine (mkl_sparse_optimize()) until an >>> operation that uses an MKL-provided kernel is requested. (You can use an >>> "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() >>> by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) >>> >>> If memory is at enough of a premium for you that you can't afford the >>> extra copy used by the MKL inspector-executor routines, then I suggest >>> using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well >>> optimized for many cases (and even has some hand-optimized kernels using >>> Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should >>> try both AIJ and AIJMKL, anyway, to see which is faster for your >>> combination of problem and computing platform. >>> >>> Best regards, >>> Richard >>> >>> On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >>> >>> We also found that if we use MatCreateSeqAIJ, then no more memory >>> increase with matrix vector multiplication. However, with >>> MatCreateMPIAIJMKL, the behavior is consistent. >>> >>> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: >>> >>>> MatCreateMPIAIJMKL >>>> >>>> parallel and sequential exhibit the same behavior. In fact, we found >>>> that doing matmult will increase the memory by the size of matrix as well. >>>> >>>> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >>>> >>>>> Karl: >>>>> What matrix format do you use? Run it in parallel or sequential? >>>>> Hong >>>>> >>>>> We used /proc/self/stat to track the resident set size during program >>>>>> run, and we saw the resident set size jumped by the size of the matrix >>>>>> right after we did matmulttranspose. >>>>>> >>>>>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >>>>>> petsc-users at mcs.anl.gov> wrote: >>>>>> >>>>>>> Kun: >>>>>>> How do you know 'MatMultTranpose creates an extra memory copy of >>>>>>> matrix'? >>>>>>> Hong >>>>>>> >>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I was using MatMultTranpose and MatMult to solver a linear system. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> However we found out, MatMultTranpose create an extra memory copy >>>>>>>> of matrix for its operation. This extra memory copy is not stated >>>>>>>> everywhere in petsc manual. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This basically double my memory requirement to solve my system. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I remember mkl?s routine can do inplace matrix transpose vector >>>>>>>> product, without transposing the matrix itself. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Is this always the case? Or there is way to make petsc to do >>>>>>>> inplace matrix transpose vector product. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Any help is greatly appreciated. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Kun >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Schlumberger-Private >>>>>>>> >>>>>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 30 18:34:34 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 30 Jul 2019 23:34:34 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> Message-ID: <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Thanks, this is enough information to diagnose the problem. The problem is that 32 bit integers are not large enough to contain the "counts", in this case the number of nonzeros in the matrix. A signed integer can only be as large as PETSC_MAX_INT 2147483647. You need to configure PETSc with the additional option --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so you don't run out space in int for such large counts. We don't do a perfect job of detecting when there is overflow of int which is why you ended up with crazy allocation requests like 18446744058325389312. I will add some more error checking to provide more useful error messages in this case. Barry The reason this worked for 4 processes is that the largest count in that case was roughly 6,653,750,976/4 which does fit into an int. PETSc only needs to know the number of nonzeros on each process, it doesn't need to know the amount across all the processors. In other words you may want to use a different PETSC_ARCH (different configuration) for small number of processors and large number depending on how large your problem is. Or you can always use 64 bit integers at a little performance and memory cost. > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users wrote: > > number of rows is 26,326,575. maximum column index is 36,416,250. number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB for coefficients in PetscScalar and column index in PetscInt. I can run the program in 4 processes with this input but not single process. Here are the snap shots of the error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [0]PETSC ERROR: Memory requested 18446744058325389312 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #2 PetscMallocA() line 390 in /petsc-3.10.4/src/sys/memory/mal.c > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #6 *** (my code) > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > > > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley wrote: > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users wrote: > Hi, Richard, > > We have a new question. Is there a limit for MatCreateMPIAIJ and MatSetValues? What I mean is that, we tried to create a sparse matrix and populate it with 50GB of data in one process, I got a crash and error saying object too big. Thank you for any insight. > > 1) Always send the complete error. > > 2) It sounds like you got an out of memory error for that process. > > Matt > > Regards, > > Karl > > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users wrote: > Hi Kun and Karl, > > If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > > The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > > If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform. > > Best regards, > Richard > > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >> We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: >> MatCreateMPIAIJMKL >> >> parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >> Karl: >> What matrix format do you use? Run it in parallel or sequential? >> Hong >> >> We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users wrote: >> Kun: >> How do you know 'MatMultTranpose creates an extra memory copy of matrix'? >> Hong >> >> Hi, >> >> >> I was using MatMultTranpose and MatMult to solver a linear system. >> >> >> However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. >> >> >> This basically double my memory requirement to solve my system. >> >> >> I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. >> >> >> Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. >> >> >> Any help is greatly appreciated. >> >> >> Regards, >> >> Kun >> >> >> >> Schlumberger-Private > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From karl.linkui at gmail.com Tue Jul 30 19:35:26 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Tue, 30 Jul 2019 19:35:26 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: Thanks for the feedback, very helpful. I have another question, when I run 4 processes, even though the matrix is only 49.7GB, I found the memory footprint of the matrix is about 52.8GB. Where does these extra memory comes from? Does MatCreateAIJ still reserves some extra memory? I thought after MatAssembly all unused space would be released but in at least one of the processes, the memory footprint of the local matrix actually increased after MatAssembly by couple of GBs. I will greatly appreciate any info. On Tue, Jul 30, 2019 at 6:34 PM Smith, Barry F. wrote: > > Thanks, this is enough information to diagnose the problem. > > The problem is that 32 bit integers are not large enough to contain the > "counts", in this case the number of nonzeros in the matrix. A signed > integer can only be as large as PETSC_MAX_INT 2147483647. > > You need to configure PETSc with the additional option > --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so > you don't run out space in int for such large counts. > > > We don't do a perfect job of detecting when there is overflow of int > which is why you ended up with crazy allocation requests like > 18446744058325389312. > > I will add some more error checking to provide more useful error > messages in this case. > > Barry > > The reason this worked for 4 processes is that the largest count in > that case was roughly 6,653,750,976/4 which does fit into an int. PETSc > only needs to know the number of nonzeros on each process, it doesn't need > to know the amount across all the processors. In other words you may want > to use a different PETSC_ARCH (different configuration) for small number of > processors and large number depending on how large your problem is. Or you > can always use 64 bit integers at a little performance and memory cost. > > > > > > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > number of rows is 26,326,575. maximum column index is 36,416,250. number > of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB for > coefficients in PetscScalar and column index in PetscInt. I can run the > program in 4 processes with this input but not single process. Here are the > snap shots of the error: > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Out of memory. This could be due to allocating > > [0]PETSC ERROR: too large an object or bleeding by not properly > > [0]PETSC ERROR: destroying unneeded objects. > > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 > > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > > [0]PETSC ERROR: Memory requested 18446744058325389312 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in > /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #2 PetscMallocA() line 390 in > /petsc-3.10.4/src/sys/memory/mal.c > > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 > in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in > /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in > /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #6 *** (my code) > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > > > > > > > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley > wrote: > > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hi, Richard, > > > > We have a new question. Is there a limit for MatCreateMPIAIJ and > MatSetValues? What I mean is that, we tried to create a sparse matrix and > populate it with 50GB of data in one process, I got a crash and error > saying object too big. Thank you for any insight. > > > > 1) Always send the complete error. > > > > 2) It sounds like you got an out of memory error for that process. > > > > Matt > > > > Regards, > > > > Karl > > > > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hi Kun and Karl, > > > > If you are using the AIJMKL matrix types and have a recent version of > MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, > which are described at > > > > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > > > > The inspector-executor analysis routines take the AIJ (compressed sparse > row) format data from PETSc and then create a copy in an optimized, > internal layout used by MKL. We have to keep PETSc's own, AIJ > representation around, as it is needed for several operations that MKL does > not provide. This does, unfortunately, mean that roughly double (or more, > depending on what MKL decides to do) the amount of memory is required. The > reason you see the memory usage increase right when a MatMult() or > MatMultTranspose() operation occurs is that the we default to a "lazy" > approach to calling the analysis routine (mkl_sparse_optimize()) until an > operation that uses an MKL-provided kernel is requested. (You can use an > "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() > by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > > > > If memory is at enough of a premium for you that you can't afford the > extra copy used by the MKL inspector-executor routines, then I suggest > using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well > optimized for many cases (and even has some hand-optimized kernels using > Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should > try both AIJ and AIJMKL, anyway, to see which is faster for your > combination of problem and computing platform. > > > > Best regards, > > Richard > > > > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: > >> We also found that if we use MatCreateSeqAIJ, then no more memory > increase with matrix vector multiplication. However, with > MatCreateMPIAIJMKL, the behavior is consistent. > >> > >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: > >> MatCreateMPIAIJMKL > >> > >> parallel and sequential exhibit the same behavior. In fact, we found > that doing matmult will increase the memory by the size of matrix as well. > >> > >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: > >> Karl: > >> What matrix format do you use? Run it in parallel or sequential? > >> Hong > >> > >> We used /proc/self/stat to track the resident set size during program > run, and we saw the resident set size jumped by the size of the matrix > right after we did matmulttranspose. > >> > >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Kun: > >> How do you know 'MatMultTranpose creates an extra memory copy of > matrix'? > >> Hong > >> > >> Hi, > >> > >> > >> I was using MatMultTranpose and MatMult to solver a linear system. > >> > >> > >> However we found out, MatMultTranpose create an extra memory copy of > matrix for its operation. This extra memory copy is not stated everywhere > in petsc manual. > >> > >> > >> This basically double my memory requirement to solve my system. > >> > >> > >> I remember mkl?s routine can do inplace matrix transpose vector > product, without transposing the matrix itself. > >> > >> > >> Is this always the case? Or there is way to make petsc to do inplace > matrix transpose vector product. > >> > >> > >> Any help is greatly appreciated. > >> > >> > >> Regards, > >> > >> Kun > >> > >> > >> > >> Schlumberger-Private > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtmills at anl.gov Tue Jul 30 20:12:55 2019 From: rtmills at anl.gov (Mills, Richard Tran) Date: Wed, 31 Jul 2019 01:12:55 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: Hi Karl, I'll let one of my colleagues who has a better understanding of exactly what happens with memory during matrix PETSc matrix assembly chime in, but let me ask how you know that the memory footprint is actually larger than you think it should be? Are you looking at the resident set size reported by a tool like 'top'? Keep in mind that even if extra buffers were allocated and then free()d, that resident set size of your process may stay the same, and only decrease when the OS's memory manager decides it really needs those pages for something else. --Richard On 7/30/19 5:35 PM, Karl Lin via petsc-users wrote: Thanks for the feedback, very helpful. I have another question, when I run 4 processes, even though the matrix is only 49.7GB, I found the memory footprint of the matrix is about 52.8GB. Where does these extra memory comes from? Does MatCreateAIJ still reserves some extra memory? I thought after MatAssembly all unused space would be released but in at least one of the processes, the memory footprint of the local matrix actually increased after MatAssembly by couple of GBs. I will greatly appreciate any info. On Tue, Jul 30, 2019 at 6:34 PM Smith, Barry F. > wrote: Thanks, this is enough information to diagnose the problem. The problem is that 32 bit integers are not large enough to contain the "counts", in this case the number of nonzeros in the matrix. A signed integer can only be as large as PETSC_MAX_INT 2147483647. You need to configure PETSc with the additional option --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so you don't run out space in int for such large counts. We don't do a perfect job of detecting when there is overflow of int which is why you ended up with crazy allocation requests like 18446744058325389312. I will add some more error checking to provide more useful error messages in this case. Barry The reason this worked for 4 processes is that the largest count in that case was roughly 6,653,750,976/4 which does fit into an int. PETSc only needs to know the number of nonzeros on each process, it doesn't need to know the amount across all the processors. In other words you may want to use a different PETSC_ARCH (different configuration) for small number of processors and large number depending on how large your problem is. Or you can always use 64 bit integers at a little performance and memory cost. > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users > wrote: > > number of rows is 26,326,575. maximum column index is 36,416,250. number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB for coefficients in PetscScalar and column index in PetscInt. I can run the program in 4 processes with this input but not single process. Here are the snap shots of the error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [0]PETSC ERROR: Memory requested 18446744058325389312 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #2 PetscMallocA() line 390 in /petsc-3.10.4/src/sys/memory/mal.c > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #6 *** (my code) > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > > > > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley > wrote: > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users > wrote: > Hi, Richard, > > We have a new question. Is there a limit for MatCreateMPIAIJ and MatSetValues? What I mean is that, we tried to create a sparse matrix and populate it with 50GB of data in one process, I got a crash and error saying object too big. Thank you for any insight. > > 1) Always send the complete error. > > 2) It sounds like you got an out of memory error for that process. > > Matt > > Regards, > > Karl > > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users > wrote: > Hi Kun and Karl, > > If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > > The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > > If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform. > > Best regards, > Richard > > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >> We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin > wrote: >> MatCreateMPIAIJMKL >> >> parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong > wrote: >> Karl: >> What matrix format do you use? Run it in parallel or sequential? >> Hong >> >> We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users > wrote: >> Kun: >> How do you know 'MatMultTranpose creates an extra memory copy of matrix'? >> Hong >> >> Hi, >> >> >> I was using MatMultTranpose and MatMult to solver a linear system. >> >> >> However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. >> >> >> This basically double my memory requirement to solve my system. >> >> >> I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. >> >> >> Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. >> >> >> Any help is greatly appreciated. >> >> >> Regards, >> >> Kun >> >> >> >> Schlumberger-Private > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Tue Jul 30 20:29:21 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Tue, 30 Jul 2019 20:29:21 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: I checked the resident set size via /proc/self/stat On Tue, Jul 30, 2019 at 8:13 PM Mills, Richard Tran via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hi Karl, > > I'll let one of my colleagues who has a better understanding of exactly > what happens with memory during matrix PETSc matrix assembly chime in, but > let me ask how you know that the memory footprint is actually larger than > you think it should be? Are you looking at the resident set size reported > by a tool like 'top'? Keep in mind that even if extra buffers were > allocated and then free()d, that resident set size of your process may stay > the same, and only decrease when the OS's memory manager decides it really > needs those pages for something else. > > --Richard > > On 7/30/19 5:35 PM, Karl Lin via petsc-users wrote: > > Thanks for the feedback, very helpful. > > I have another question, when I run 4 processes, even though the matrix is > only 49.7GB, I found the memory footprint of the matrix is about 52.8GB. > Where does these extra memory comes from? Does MatCreateAIJ still reserves > some extra memory? I thought after MatAssembly all unused space would be > released but in at least one of the processes, the memory footprint of the > local matrix actually increased after MatAssembly by couple of GBs. I will > greatly appreciate any info. > > > On Tue, Jul 30, 2019 at 6:34 PM Smith, Barry F. > wrote: > >> >> Thanks, this is enough information to diagnose the problem. >> >> The problem is that 32 bit integers are not large enough to contain >> the "counts", in this case the number of nonzeros in the matrix. A signed >> integer can only be as large as PETSC_MAX_INT 2147483647. >> >> You need to configure PETSc with the additional option >> --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so >> you don't run out space in int for such large counts. >> >> >> We don't do a perfect job of detecting when there is overflow of int >> which is why you ended up with crazy allocation requests like >> 18446744058325389312. >> >> I will add some more error checking to provide more useful error >> messages in this case. >> >> Barry >> >> The reason this worked for 4 processes is that the largest count in >> that case was roughly 6,653,750,976/4 which does fit into an int. PETSc >> only needs to know the number of nonzeros on each process, it doesn't need >> to know the amount across all the processors. In other words you may want >> to use a different PETSC_ARCH (different configuration) for small number of >> processors and large number depending on how large your problem is. Or you >> can always use 64 bit integers at a little performance and memory cost. >> >> >> >> >> > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > >> > number of rows is 26,326,575. maximum column index is 36,416,250. >> number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB >> for coefficients in PetscScalar and column index in PetscInt. I can run the >> program in 4 processes with this input but not single process. Here are the >> snap shots of the error: >> > >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Out of memory. This could be due to allocating >> > [0]PETSC ERROR: too large an object or bleeding by not properly >> > [0]PETSC ERROR: destroying unneeded objects. >> > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 >> > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. >> > [0]PETSC ERROR: Memory requested 18446744058325389312 >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> > >> > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in >> /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #2 PetscMallocA() line 390 in >> /petsc-3.10.4/src/sys/memory/mal.c >> > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 >> in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in >> /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in >> /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c >> > [0]PETSC ERROR: #6 *** (my code) >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > [0]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X to find memory corruption errors >> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> > [0]PETSC ERROR: to get more information on the crash. >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Signal received >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> > >> > >> > >> > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley >> wrote: >> > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > Hi, Richard, >> > >> > We have a new question. Is there a limit for MatCreateMPIAIJ and >> MatSetValues? What I mean is that, we tried to create a sparse matrix and >> populate it with 50GB of data in one process, I got a crash and error >> saying object too big. Thank you for any insight. >> > >> > 1) Always send the complete error. >> > >> > 2) It sounds like you got an out of memory error for that process. >> > >> > Matt >> > >> > Regards, >> > >> > Karl >> > >> > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > Hi Kun and Karl, >> > >> > If you are using the AIJMKL matrix types and have a recent version of >> MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, >> which are described at >> > >> > >> https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines >> > >> > The inspector-executor analysis routines take the AIJ (compressed >> sparse row) format data from PETSc and then create a copy in an optimized, >> internal layout used by MKL. We have to keep PETSc's own, AIJ >> representation around, as it is needed for several operations that MKL does >> not provide. This does, unfortunately, mean that roughly double (or more, >> depending on what MKL decides to do) the amount of memory is required. The >> reason you see the memory usage increase right when a MatMult() or >> MatMultTranspose() operation occurs is that the we default to a "lazy" >> approach to calling the analysis routine (mkl_sparse_optimize()) until an >> operation that uses an MKL-provided kernel is requested. (You can use an >> "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() >> by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) >> > >> > If memory is at enough of a premium for you that you can't afford the >> extra copy used by the MKL inspector-executor routines, then I suggest >> using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well >> optimized for many cases (and even has some hand-optimized kernels using >> Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should >> try both AIJ and AIJMKL, anyway, to see which is faster for your >> combination of problem and computing platform. >> > >> > Best regards, >> > Richard >> > >> > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >> >> We also found that if we use MatCreateSeqAIJ, then no more memory >> increase with matrix vector multiplication. However, with >> MatCreateMPIAIJMKL, the behavior is consistent. >> >> >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin >> wrote: >> >> MatCreateMPIAIJMKL >> >> >> >> parallel and sequential exhibit the same behavior. In fact, we found >> that doing matmult will increase the memory by the size of matrix as well. >> >> >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong >> wrote: >> >> Karl: >> >> What matrix format do you use? Run it in parallel or sequential? >> >> Hong >> >> >> >> We used /proc/self/stat to track the resident set size during program >> run, and we saw the resident set size jumped by the size of the matrix >> right after we did matmulttranspose. >> >> >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> >> Kun: >> >> How do you know 'MatMultTranpose creates an extra memory copy of >> matrix'? >> >> Hong >> >> >> >> Hi, >> >> >> >> >> >> I was using MatMultTranpose and MatMult to solver a linear system. >> >> >> >> >> >> However we found out, MatMultTranpose create an extra memory copy of >> matrix for its operation. This extra memory copy is not stated everywhere >> in petsc manual. >> >> >> >> >> >> This basically double my memory requirement to solve my system. >> >> >> >> >> >> I remember mkl?s routine can do inplace matrix transpose vector >> product, without transposing the matrix itself. >> >> >> >> >> >> Is this always the case? Or there is way to make petsc to do inplace >> matrix transpose vector product. >> >> >> >> >> >> Any help is greatly appreciated. >> >> >> >> >> >> Regards, >> >> >> >> Kun >> >> >> >> >> >> >> >> Schlumberger-Private >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gohardoust at gmail.com Tue Jul 30 14:03:22 2019 From: gohardoust at gmail.com (Mohammad Gohardoust) Date: Tue, 30 Jul 2019 19:03:22 +0000 Subject: [petsc-users] MINRES accuracy for a SPD matrix Message-ID: Hi, I am trying to evaluate different solver/preconditioner combinations for solving Richards' equation in porous media. The coefficient matrix is symmetric positive definite and is usually solved by CG solver. I tried MINRES with various preconditioners. The solution is acceptable but with somehow considerable errors (compared to other solvers) which requires me to decrease the tolerance threshold in my Picard iteration leading to larger computation time. I would like to know if this is common for MINRES and how to possibly overcome this? Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 30 21:16:31 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jul 2019 02:16:31 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: <03CFBD36-DAED-44C0-807E-04A7BB80646F@anl.gov> https://bitbucket.org/petsc/petsc/pull-requests/1924/add-petscintcast-to-check-for-integer/diff > On Jul 30, 2019, at 6:34 PM, Smith, Barry F. wrote: > > > Thanks, this is enough information to diagnose the problem. > > The problem is that 32 bit integers are not large enough to contain the "counts", in this case the number of nonzeros in the matrix. A signed integer can only be as large as PETSC_MAX_INT 2147483647. > > You need to configure PETSc with the additional option --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so you don't run out space in int for such large counts. > > > We don't do a perfect job of detecting when there is overflow of int which is why you ended up with crazy allocation requests like 18446744058325389312. > > I will add some more error checking to provide more useful error messages in this case. > > Barry > > The reason this worked for 4 processes is that the largest count in that case was roughly 6,653,750,976/4 which does fit into an int. PETSc only needs to know the number of nonzeros on each process, it doesn't need to know the amount across all the processors. In other words you may want to use a different PETSC_ARCH (different configuration) for small number of processors and large number depending on how large your problem is. Or you can always use 64 bit integers at a little performance and memory cost. > > > > >> On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users wrote: >> >> number of rows is 26,326,575. maximum column index is 36,416,250. number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB for coefficients in PetscScalar and column index in PetscInt. I can run the program in 4 processes with this input but not single process. Here are the snap shots of the error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Out of memory. This could be due to allocating >> [0]PETSC ERROR: too large an object or bleeding by not properly >> [0]PETSC ERROR: destroying unneeded objects. >> [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 >> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. >> [0]PETSC ERROR: Memory requested 18446744058325389312 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> >> [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: #2 PetscMallocA() line 390 in /petsc-3.10.4/src/sys/memory/mal.c >> [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #6 *** (my code) >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> >> >> >> On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley wrote: >> On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users wrote: >> Hi, Richard, >> >> We have a new question. Is there a limit for MatCreateMPIAIJ and MatSetValues? What I mean is that, we tried to create a sparse matrix and populate it with 50GB of data in one process, I got a crash and error saying object too big. Thank you for any insight. >> >> 1) Always send the complete error. >> >> 2) It sounds like you got an out of memory error for that process. >> >> Matt >> >> Regards, >> >> Karl >> >> On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users wrote: >> Hi Kun and Karl, >> >> If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at >> >> https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines >> >> The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) >> >> If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform. >> >> Best regards, >> Richard >> >> On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >>> We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. >>> >>> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: >>> MatCreateMPIAIJMKL >>> >>> parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. >>> >>> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >>> Karl: >>> What matrix format do you use? Run it in parallel or sequential? >>> Hong >>> >>> We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. >>> >>> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users wrote: >>> Kun: >>> How do you know 'MatMultTranpose creates an extra memory copy of matrix'? >>> Hong >>> >>> Hi, >>> >>> >>> I was using MatMultTranpose and MatMult to solver a linear system. >>> >>> >>> However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. >>> >>> >>> This basically double my memory requirement to solve my system. >>> >>> >>> I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. >>> >>> >>> Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. >>> >>> >>> Any help is greatly appreciated. >>> >>> >>> Regards, >>> >>> Kun >>> >>> >>> >>> Schlumberger-Private >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > From jed at jedbrown.org Tue Jul 30 22:19:44 2019 From: jed at jedbrown.org (Jed Brown) Date: Tue, 30 Jul 2019 21:19:44 -0600 Subject: [petsc-users] MINRES accuracy for a SPD matrix In-Reply-To: References: Message-ID: <87a7cu51rj.fsf@jedbrown.org> Are you saying that the MINRES error is larger than CG error? In which norm? And which norm are you using for CG? (Output from -ksp_monitor_true_residual -ksp_view would be useful.) CG does find a solution that is optimal in a different inner product, though this is usually pretty benign unless you have a poor preconditioner. Mohammad Gohardoust via petsc-users writes: > Hi, > > I am trying to evaluate different solver/preconditioner combinations for > solving Richards' equation in porous media. The coefficient matrix is > symmetric positive definite and is usually solved by CG solver. I tried > MINRES with various preconditioners. The solution is acceptable but with > somehow considerable errors (compared to other solvers) which requires me > to decrease the tolerance threshold in my Picard iteration leading to > larger computation time. I would like to know if this is common for MINRES > and how to possibly overcome this? > > Thanks, > Mohammad From bsmith at mcs.anl.gov Wed Jul 31 04:11:43 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jul 2019 09:11:43 +0000 Subject: [petsc-users] MINRES accuracy for a SPD matrix In-Reply-To: References: Message-ID: <52F58D8F-BE51-4E03-8577-B1C22C5BD653@anl.gov> You could run with -ksp_view_mat binary:afilename -ksp_view_rhs binary:afilename and email the file afilename to petsc-maint at mcs.anl.gov with the options you've used for a particularly clear case and we can take a look at it. Conventional wisdom is for SPD matrices CG is slightly better than MINRES but that is only conventional wisdom. Barry > On Jul 30, 2019, at 2:03 PM, Mohammad Gohardoust via petsc-users wrote: > > Hi, > > I am trying to evaluate different solver/preconditioner combinations for solving Richards' equation in porous media. The coefficient matrix is symmetric positive definite and is usually solved by CG solver. I tried MINRES with various preconditioners. The solution is acceptable but with somehow considerable errors (compared to other solvers) which requires me to decrease the tolerance threshold in my Picard iteration leading to larger computation time. I would like to know if this is common for MINRES and how to possibly overcome this? > > Thanks, > Mohammad From bsmith at mcs.anl.gov Wed Jul 31 04:34:25 2019 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jul 2019 09:34:25 +0000 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: You can run with -log_view -log_view_memory to get a summary that contains information in the right columns about the amounts of memory allocated within each method. So for example the line with MatAssemblyEnd may tell you something. One additional "chunk" of memory that is needed in the parallel case but not the sequential case is 1) a vector whose length is the number of columns of the "off-diagonal" part of the parallel matrix (see the explanation of diagonal and off-diagonal in the manual page for https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html) and a VecScatter that is needed to communicate these "off-diagonal" vector parts. Normally this is pretty small compared to the memory of the matrix but it can be measurable depending on the parallel partitioning of the matrix and the nonzero structure fo the matrix (for example with a stencil width of 2 this extra memory will be larger than a stencil width of 1). Barry > On Jul 30, 2019, at 8:29 PM, Karl Lin via petsc-users wrote: > > I checked the resident set size via /proc/self/stat > > On Tue, Jul 30, 2019 at 8:13 PM Mills, Richard Tran via petsc-users wrote: > Hi Karl, > > I'll let one of my colleagues who has a better understanding of exactly what happens with memory during matrix PETSc matrix assembly chime in, but let me ask how you know that the memory footprint is actually larger than you think it should be? Are you looking at the resident set size reported by a tool like 'top'? Keep in mind that even if extra buffers were allocated and then free()d, that resident set size of your process may stay the same, and only decrease when the OS's memory manager decides it really needs those pages for something else. > > --Richard > > On 7/30/19 5:35 PM, Karl Lin via petsc-users wrote: >> Thanks for the feedback, very helpful. >> >> I have another question, when I run 4 processes, even though the matrix is only 49.7GB, I found the memory footprint of the matrix is about 52.8GB. Where does these extra memory comes from? Does MatCreateAIJ still reserves some extra memory? I thought after MatAssembly all unused space would be released but in at least one of the processes, the memory footprint of the local matrix actually increased after MatAssembly by couple of GBs. I will greatly appreciate any info. >> >> >> On Tue, Jul 30, 2019 at 6:34 PM Smith, Barry F. wrote: >> >> Thanks, this is enough information to diagnose the problem. >> >> The problem is that 32 bit integers are not large enough to contain the "counts", in this case the number of nonzeros in the matrix. A signed integer can only be as large as PETSC_MAX_INT 2147483647. >> >> You need to configure PETSc with the additional option --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so you don't run out space in int for such large counts. >> >> >> We don't do a perfect job of detecting when there is overflow of int which is why you ended up with crazy allocation requests like 18446744058325389312. >> >> I will add some more error checking to provide more useful error messages in this case. >> >> Barry >> >> The reason this worked for 4 processes is that the largest count in that case was roughly 6,653,750,976/4 which does fit into an int. PETSc only needs to know the number of nonzeros on each process, it doesn't need to know the amount across all the processors. In other words you may want to use a different PETSC_ARCH (different configuration) for small number of processors and large number depending on how large your problem is. Or you can always use 64 bit integers at a little performance and memory cost. >> >> >> >> >> > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users wrote: >> > >> > number of rows is 26,326,575. maximum column index is 36,416,250. number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB for coefficients in PetscScalar and column index in PetscInt. I can run the program in 4 processes with this input but not single process. Here are the snap shots of the error: >> > >> > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> > [0]PETSC ERROR: Out of memory. This could be due to allocating >> > [0]PETSC ERROR: too large an object or bleeding by not properly >> > [0]PETSC ERROR: destroying unneeded objects. >> > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 >> > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. >> > [0]PETSC ERROR: Memory requested 18446744058325389312 >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> > >> > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #2 PetscMallocA() line 390 in /petsc-3.10.4/src/sys/memory/mal.c >> > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in /petsc-3.10.4/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c >> > [0]PETSC ERROR: #6 *** (my code) >> > [0]PETSC ERROR: ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> > [0]PETSC ERROR: to get more information on the crash. >> > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> > [0]PETSC ERROR: Signal received >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 >> > >> > >> > >> > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley wrote: >> > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users wrote: >> > Hi, Richard, >> > >> > We have a new question. Is there a limit for MatCreateMPIAIJ and MatSetValues? What I mean is that, we tried to create a sparse matrix and populate it with 50GB of data in one process, I got a crash and error saying object too big. Thank you for any insight. >> > >> > 1) Always send the complete error. >> > >> > 2) It sounds like you got an out of memory error for that process. >> > >> > Matt >> > >> > Regards, >> > >> > Karl >> > >> > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users wrote: >> > Hi Kun and Karl, >> > >> > If you are using the AIJMKL matrix types and have a recent version of MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, which are described at >> > >> > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines >> > >> > The inspector-executor analysis routines take the AIJ (compressed sparse row) format data from PETSc and then create a copy in an optimized, internal layout used by MKL. We have to keep PETSc's own, AIJ representation around, as it is needed for several operations that MKL does not provide. This does, unfortunately, mean that roughly double (or more, depending on what MKL decides to do) the amount of memory is required. The reason you see the memory usage increase right when a MatMult() or MatMultTranspose() operation occurs is that the we default to a "lazy" approach to calling the analysis routine (mkl_sparse_optimize()) until an operation that uses an MKL-provided kernel is requested. (You can use an "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) >> > >> > If memory is at enough of a premium for you that you can't afford the extra copy used by the MKL inspector-executor routines, then I suggest using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well optimized for many cases (and even has some hand-optimized kernels using Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should try both AIJ and AIJMKL, anyway, to see which is faster for your combination of problem and computing platform. >> > >> > Best regards, >> > Richard >> > >> > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: >> >> We also found that if we use MatCreateSeqAIJ, then no more memory increase with matrix vector multiplication. However, with MatCreateMPIAIJMKL, the behavior is consistent. >> >> >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin wrote: >> >> MatCreateMPIAIJMKL >> >> >> >> parallel and sequential exhibit the same behavior. In fact, we found that doing matmult will increase the memory by the size of matrix as well. >> >> >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong wrote: >> >> Karl: >> >> What matrix format do you use? Run it in parallel or sequential? >> >> Hong >> >> >> >> We used /proc/self/stat to track the resident set size during program run, and we saw the resident set size jumped by the size of the matrix right after we did matmulttranspose. >> >> >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users wrote: >> >> Kun: >> >> How do you know 'MatMultTranpose creates an extra memory copy of matrix'? >> >> Hong >> >> >> >> Hi, >> >> >> >> >> >> I was using MatMultTranpose and MatMult to solver a linear system. >> >> >> >> >> >> However we found out, MatMultTranpose create an extra memory copy of matrix for its operation. This extra memory copy is not stated everywhere in petsc manual. >> >> >> >> >> >> This basically double my memory requirement to solve my system. >> >> >> >> >> >> I remember mkl?s routine can do inplace matrix transpose vector product, without transposing the matrix itself. >> >> >> >> >> >> Is this always the case? Or there is way to make petsc to do inplace matrix transpose vector product. >> >> >> >> >> >> Any help is greatly appreciated. >> >> >> >> >> >> Regards, >> >> >> >> Kun >> >> >> >> >> >> >> >> Schlumberger-Private >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> > From karl.linkui at gmail.com Wed Jul 31 07:34:32 2019 From: karl.linkui at gmail.com (Karl Lin) Date: Wed, 31 Jul 2019 07:34:32 -0500 Subject: [petsc-users] MatMultTranspose memory usage In-Reply-To: References: <46ee30b7-116b-7d67-d5eb-66ae96f0e174@anl.gov> <6728C71A-2937-4341-A52D-E28CB3669183@anl.gov> Message-ID: Thanks Barry. I will run with log view to see the info. On Wed, Jul 31, 2019 at 4:34 AM Smith, Barry F. wrote: > > You can run with -log_view -log_view_memory to get a summary that > contains information in the right columns about the amounts of memory > allocated within each method. So for example the line with MatAssemblyEnd > may tell you something. > > One additional "chunk" of memory that is needed in the parallel case but > not the sequential case is 1) a vector whose length is the number of > columns of the "off-diagonal" part of the parallel matrix (see the > explanation of diagonal and off-diagonal in the manual page for > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html) > and a VecScatter that is needed to communicate these "off-diagonal" vector > parts. Normally this is pretty small compared to the memory of the matrix > but it can be measurable depending on the parallel partitioning of the > matrix and the nonzero structure fo the matrix (for example with a stencil > width of 2 this extra memory will be larger than a stencil width of 1). > > Barry > > > > On Jul 30, 2019, at 8:29 PM, Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > I checked the resident set size via /proc/self/stat > > > > On Tue, Jul 30, 2019 at 8:13 PM Mills, Richard Tran via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Hi Karl, > > > > I'll let one of my colleagues who has a better understanding of exactly > what happens with memory during matrix PETSc matrix assembly chime in, but > let me ask how you know that the memory footprint is actually larger than > you think it should be? Are you looking at the resident set size reported > by a tool like 'top'? Keep in mind that even if extra buffers were > allocated and then free()d, that resident set size of your process may stay > the same, and only decrease when the OS's memory manager decides it really > needs those pages for something else. > > > > --Richard > > > > On 7/30/19 5:35 PM, Karl Lin via petsc-users wrote: > >> Thanks for the feedback, very helpful. > >> > >> I have another question, when I run 4 processes, even though the matrix > is only 49.7GB, I found the memory footprint of the matrix is about 52.8GB. > Where does these extra memory comes from? Does MatCreateAIJ still reserves > some extra memory? I thought after MatAssembly all unused space would be > released but in at least one of the processes, the memory footprint of the > local matrix actually increased after MatAssembly by couple of GBs. I will > greatly appreciate any info. > >> > >> > >> On Tue, Jul 30, 2019 at 6:34 PM Smith, Barry F. > wrote: > >> > >> Thanks, this is enough information to diagnose the problem. > >> > >> The problem is that 32 bit integers are not large enough to contain > the "counts", in this case the number of nonzeros in the matrix. A signed > integer can only be as large as PETSC_MAX_INT 2147483647. > >> > >> You need to configure PETSc with the additional option > --with-64-bit-indices, then PETSc will use 64 bit integers for PetscInt so > you don't run out space in int for such large counts. > >> > >> > >> We don't do a perfect job of detecting when there is overflow of > int which is why you ended up with crazy allocation requests like > 18446744058325389312. > >> > >> I will add some more error checking to provide more useful error > messages in this case. > >> > >> Barry > >> > >> The reason this worked for 4 processes is that the largest count in > that case was roughly 6,653,750,976/4 which does fit into an int. PETSc > only needs to know the number of nonzeros on each process, it doesn't need > to know the amount across all the processors. In other words you may want > to use a different PETSC_ARCH (different configuration) for small number of > processors and large number depending on how large your problem is. Or you > can always use 64 bit integers at a little performance and memory cost. > >> > >> > >> > >> > >> > On Jul 30, 2019, at 5:27 PM, Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> > > >> > number of rows is 26,326,575. maximum column index is 36,416,250. > number of nonzero coefficients is 6,653,750,976, which amounts to 49.7GB > for coefficients in PetscScalar and column index in PetscInt. I can run the > program in 4 processes with this input but not single process. Here are the > snap shots of the error: > >> > > >> > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >> > [0]PETSC ERROR: Out of memory. This could be due to allocating > >> > [0]PETSC ERROR: too large an object or bleeding by not properly > >> > [0]PETSC ERROR: destroying unneeded objects. > >> > [0]PETSC ERROR: Memory allocated 0 Memory used by process 727035904 > >> > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > >> > [0]PETSC ERROR: Memory requested 18446744058325389312 > >> > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > >> > > >> > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 in > /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > >> > [0]PETSC ERROR: #2 PetscMallocA() line 390 in > /petsc-3.10.4/src/sys/memory/mal.c > >> > [0]PETSC ERROR: #3 MatSeqAIJSetPreallocation_SeqAIJ() line 3711 > in/petsc-3.10.4/src/mat/impls/aij/seq/aij.c > >> > [0]PETSC ERROR: #4 MatSeqAIJSetPreallocation() line 3649 in > /petsc-3.10.4/src/mat/impls/aij/seq/aij.c > >> > [0]PETSC ERROR: #5 MatCreateAIJ() line 4413 in > /petsc-3.10.4/src/mat/impls/aij/mpi/mpiaij.c > >> > [0]PETSC ERROR: #6 *** (my code) > >> > [0]PETSC ERROR: > ------------------------------------------------------------------------ > >> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > >> > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > >> > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > >> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, > link, and run > >> > [0]PETSC ERROR: to get more information on the crash. > >> > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >> > [0]PETSC ERROR: Signal received > >> > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > >> > [0]PETSC ERROR: Petsc Release Version 3.10.4, Feb, 26, 2019 > >> > > >> > > >> > > >> > On Tue, Jul 30, 2019 at 10:34 AM Matthew Knepley > wrote: > >> > On Wed, Jul 31, 2019 at 3:25 AM Karl Lin via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> > Hi, Richard, > >> > > >> > We have a new question. Is there a limit for MatCreateMPIAIJ and > MatSetValues? What I mean is that, we tried to create a sparse matrix and > populate it with 50GB of data in one process, I got a crash and error > saying object too big. Thank you for any insight. > >> > > >> > 1) Always send the complete error. > >> > > >> > 2) It sounds like you got an out of memory error for that process. > >> > > >> > Matt > >> > > >> > Regards, > >> > > >> > Karl > >> > > >> > On Thu, Jul 18, 2019 at 2:36 PM Mills, Richard Tran via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> > Hi Kun and Karl, > >> > > >> > If you are using the AIJMKL matrix types and have a recent version of > MKL, the AIJMKL code uses MKL's inspector-executor sparse BLAS routines, > which are described at > >> > > >> > > https://software.intel.com/en-us/mkl-developer-reference-c-inspector-executor-sparse-blas-routines > >> > > >> > The inspector-executor analysis routines take the AIJ (compressed > sparse row) format data from PETSc and then create a copy in an optimized, > internal layout used by MKL. We have to keep PETSc's own, AIJ > representation around, as it is needed for several operations that MKL does > not provide. This does, unfortunately, mean that roughly double (or more, > depending on what MKL decides to do) the amount of memory is required. The > reason you see the memory usage increase right when a MatMult() or > MatMultTranspose() operation occurs is that the we default to a "lazy" > approach to calling the analysis routine (mkl_sparse_optimize()) until an > operation that uses an MKL-provided kernel is requested. (You can use an > "eager" approach that calls mkl_sparse_optimize() during MatAssemblyEnd() > by specifying "-mat_aijmkl_eager_inspection" in the PETSc options.) > >> > > >> > If memory is at enough of a premium for you that you can't afford the > extra copy used by the MKL inspector-executor routines, then I suggest > using the usual PETSc AIJ format instead of AIJMKL. AIJ is fairly well > optimized for many cases (and even has some hand-optimized kernels using > Intel AVX/AVX2/AVX-512 intrinsics) and often outperforms AIJMKL. You should > try both AIJ and AIJMKL, anyway, to see which is faster for your > combination of problem and computing platform. > >> > > >> > Best regards, > >> > Richard > >> > > >> > On 7/17/19 8:46 PM, Karl Lin via petsc-users wrote: > >> >> We also found that if we use MatCreateSeqAIJ, then no more memory > increase with matrix vector multiplication. However, with > MatCreateMPIAIJMKL, the behavior is consistent. > >> >> > >> >> On Wed, Jul 17, 2019 at 5:26 PM Karl Lin > wrote: > >> >> MatCreateMPIAIJMKL > >> >> > >> >> parallel and sequential exhibit the same behavior. In fact, we found > that doing matmult will increase the memory by the size of matrix as well. > >> >> > >> >> On Wed, Jul 17, 2019 at 4:55 PM Zhang, Hong > wrote: > >> >> Karl: > >> >> What matrix format do you use? Run it in parallel or sequential? > >> >> Hong > >> >> > >> >> We used /proc/self/stat to track the resident set size during > program run, and we saw the resident set size jumped by the size of the > matrix right after we did matmulttranspose. > >> >> > >> >> On Wed, Jul 17, 2019 at 12:04 PM hong--- via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> >> Kun: > >> >> How do you know 'MatMultTranpose creates an extra memory copy of > matrix'? > >> >> Hong > >> >> > >> >> Hi, > >> >> > >> >> > >> >> I was using MatMultTranpose and MatMult to solver a linear system. > >> >> > >> >> > >> >> However we found out, MatMultTranpose create an extra memory copy of > matrix for its operation. This extra memory copy is not stated everywhere > in petsc manual. > >> >> > >> >> > >> >> This basically double my memory requirement to solve my system. > >> >> > >> >> > >> >> I remember mkl?s routine can do inplace matrix transpose vector > product, without transposing the matrix itself. > >> >> > >> >> > >> >> Is this always the case? Or there is way to make petsc to do inplace > matrix transpose vector product. > >> >> > >> >> > >> >> Any help is greatly appreciated. > >> >> > >> >> > >> >> Regards, > >> >> > >> >> Kun > >> >> > >> >> > >> >> > >> >> Schlumberger-Private > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gohardoust at gmail.com Wed Jul 31 08:30:25 2019 From: gohardoust at gmail.com (Mohammad Gohardoust) Date: Wed, 31 Jul 2019 13:30:25 +0000 Subject: [petsc-users] MINRES accuracy for a SPD matrix In-Reply-To: <87a7cu51rj.fsf@jedbrown.org> References: <87a7cu51rj.fsf@jedbrown.org> Message-ID: Thanks Jed, In the code I have it took five (Picard) iterations for single time-step and here are the outputs from "ksp_monitor_true_residual -ksp_view" for one iteration: CG with bjacobi: 0 KSP preconditioned resid norm 1.084140086188e+03 true resid norm 7.471524858215e-01 ||r(i)||/||b|| 1.114656457671e-37 1 KSP preconditioned resid norm 6.296985199017e+02 true resid norm 6.804895514298e-01 ||r(i)||/||b|| 1.015203840277e-37 2 KSP preconditioned resid norm 2.107325578297e+02 true resid norm 3.137762047326e-01 ||r(i)||/||b|| 4.681141795093e-38 3 KSP preconditioned resid norm 4.225436610327e+01 true resid norm 1.732061654699e-01 ||r(i)||/||b|| 2.584015639554e-38 4 KSP preconditioned resid norm 3.812640492590e+01 true resid norm 1.011864685639e-01 ||r(i)||/||b|| 1.509573383666e-38 5 KSP preconditioned resid norm 1.883814249049e+01 true resid norm 5.398301531955e-02 ||r(i)||/||b|| 8.053579125059e-39 6 KSP preconditioned resid norm 6.625289830879e+00 true resid norm 3.680797942761e-02 ||r(i)||/||b|| 5.491282267192e-39 7 KSP preconditioned resid norm 7.791516954461e+00 true resid norm 2.236588077783e-02 ||r(i)||/||b|| 3.336704878000e-39 8 KSP preconditioned resid norm 2.968740859730e+00 true resid norm 1.501379468210e-02 ||r(i)||/||b|| 2.239867164217e-39 9 KSP preconditioned resid norm 3.027103876686e+00 true resid norm 1.041339881198e-02 ||r(i)||/||b|| 1.553546625669e-39 10 KSP preconditioned resid norm 1.702923094998e+00 true resid norm 5.758722184966e-03 ||r(i)||/||b|| 8.591280887390e-40 11 KSP preconditioned resid norm 6.959360907334e-01 true resid norm 3.209679120577e-03 ||r(i)||/||b|| 4.788432919244e-40 12 KSP preconditioned resid norm 5.947063495977e-01 true resid norm 1.792219139479e-03 ||r(i)||/||b|| 2.673762953736e-40 13 KSP preconditioned resid norm 1.638992222627e-01 true resid norm 9.582599930477e-04 ||r(i)||/||b|| 1.429602001797e-40 14 KSP preconditioned resid norm 1.619641126005e-01 true resid norm 5.129340629231e-04 ||r(i)||/||b|| 7.652323674838e-41 15 KSP preconditioned resid norm 6.298555854405e-02 true resid norm 2.630662976191e-04 ||r(i)||/||b|| 3.924614493042e-41 16 KSP preconditioned resid norm 3.381480328186e-02 true resid norm 1.405167435648e-04 ||r(i)||/||b|| 2.096331051529e-41 17 KSP preconditioned resid norm 2.188634069174e-02 true resid norm 7.170524515175e-05 ||r(i)||/||b|| 1.069751035753e-41 18 KSP preconditioned resid norm 6.591386695704e-03 true resid norm 3.789193478287e-05 ||r(i)||/||b|| 5.652994616344e-42 19 KSP preconditioned resid norm 5.988963507321e-03 true resid norm 1.922304552896e-05 ||r(i)||/||b|| 2.867833841361e-42 20 KSP preconditioned resid norm 1.890309986311e-03 true resid norm 9.615814496304e-06 ||r(i)||/||b|| 1.434557192470e-42 21 KSP preconditioned resid norm 1.322603958495e-03 true resid norm 4.933661540922e-06 ||r(i)||/||b|| 7.360395368965e-43 22 KSP preconditioned resid norm 6.147516465031e-04 true resid norm 2.434568419681e-06 ||r(i)||/||b|| 3.632066361467e-43 23 KSP preconditioned resid norm 2.443285908381e-04 true resid norm 1.258673156233e-06 ||r(i)||/||b|| 1.877780223336e-43 24 KSP preconditioned resid norm 1.903790820937e-04 true resid norm 6.814992942512e-07 ||r(i)||/||b|| 1.016710248109e-43 25 KSP preconditioned resid norm 5.897463754127e-05 true resid norm 3.612787156069e-07 ||r(i)||/||b|| 5.389818825638e-44 KSP Object: 1 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-10, divergence=1e+300 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: bjacobi number of blocks = 1 Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (sub_) 1 MPI processes type: ilu out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1., needed 1. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 package used to perform factorization: petsc total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines MINRES with bjacobi: 0 KSP preconditioned resid norm 1.400513201683e+03 true resid norm 7.911033287257e+31 ||r(i)||/||b|| 1.181672927444e-05 1 KSP preconditioned resid norm 2.320798481784e-13 true resid norm 5.094603145770e-03 ||r(i)||/||b|| 7.609820860095e-40 KSP Object: 1 MPI processes type: minres maximum iterations=10000, nonzero initial guess tolerances: relative=1e-10, absolute=1e-10, divergence=1e+300 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: bjacobi number of blocks = 1 Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (sub_) 1 MPI processes type: ilu out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1., needed 1. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 package used to perform factorization: petsc total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=408848, cols=408848 total: nonzeros=5918082, allocated nonzeros=5918082 total number of mallocs used during MatSetValues calls =0 not using I-node routines On Wed, Jul 31, 2019 at 3:19 AM Jed Brown wrote: > Are you saying that the MINRES error is larger than CG error? In which > norm? And which norm are you using for CG? (Output from > -ksp_monitor_true_residual -ksp_view would be useful.) > > CG does find a solution that is optimal in a different inner product, > though this is usually pretty benign unless you have a poor > preconditioner. > > Mohammad Gohardoust via petsc-users writes: > > > Hi, > > > > I am trying to evaluate different solver/preconditioner combinations for > > solving Richards' equation in porous media. The coefficient matrix is > > symmetric positive definite and is usually solved by CG solver. I tried > > MINRES with various preconditioners. The solution is acceptable but with > > somehow considerable errors (compared to other solvers) which requires me > > to decrease the tolerance threshold in my Picard iteration leading to > > larger computation time. I would like to know if this is common for > MINRES > > and how to possibly overcome this? > > > > Thanks, > > Mohammad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 31 15:51:19 2019 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Aug 2019 08:51:19 +1200 Subject: [petsc-users] MINRES accuracy for a SPD matrix In-Reply-To: References: <87a7cu51rj.fsf@jedbrown.org> Message-ID: On Thu, Aug 1, 2019 at 8:29 AM Mohammad Gohardoust via petsc-users < petsc-users at mcs.anl.gov> wrote: > Thanks Jed, > > In the code I have it took five (Picard) iterations for single time-step > and here are the outputs from "ksp_monitor_true_residual -ksp_view" for one > iteration: > I think there is a problem in your code. The "true residual" for step 0 should be identical since it ignores the preconditioner, but it is crazy for the MINRES case. Thanks, Matt > CG with bjacobi: > > 0 KSP preconditioned resid norm 1.084140086188e+03 true resid norm > 7.471524858215e-01 ||r(i)||/||b|| 1.114656457671e-37 > 1 KSP preconditioned resid norm 6.296985199017e+02 true resid norm > 6.804895514298e-01 ||r(i)||/||b|| 1.015203840277e-37 > 2 KSP preconditioned resid norm 2.107325578297e+02 true resid norm > 3.137762047326e-01 ||r(i)||/||b|| 4.681141795093e-38 > 3 KSP preconditioned resid norm 4.225436610327e+01 true resid norm > 1.732061654699e-01 ||r(i)||/||b|| 2.584015639554e-38 > 4 KSP preconditioned resid norm 3.812640492590e+01 true resid norm > 1.011864685639e-01 ||r(i)||/||b|| 1.509573383666e-38 > 5 KSP preconditioned resid norm 1.883814249049e+01 true resid norm > 5.398301531955e-02 ||r(i)||/||b|| 8.053579125059e-39 > 6 KSP preconditioned resid norm 6.625289830879e+00 true resid norm > 3.680797942761e-02 ||r(i)||/||b|| 5.491282267192e-39 > 7 KSP preconditioned resid norm 7.791516954461e+00 true resid norm > 2.236588077783e-02 ||r(i)||/||b|| 3.336704878000e-39 > 8 KSP preconditioned resid norm 2.968740859730e+00 true resid norm > 1.501379468210e-02 ||r(i)||/||b|| 2.239867164217e-39 > 9 KSP preconditioned resid norm 3.027103876686e+00 true resid norm > 1.041339881198e-02 ||r(i)||/||b|| 1.553546625669e-39 > 10 KSP preconditioned resid norm 1.702923094998e+00 true resid norm > 5.758722184966e-03 ||r(i)||/||b|| 8.591280887390e-40 > 11 KSP preconditioned resid norm 6.959360907334e-01 true resid norm > 3.209679120577e-03 ||r(i)||/||b|| 4.788432919244e-40 > 12 KSP preconditioned resid norm 5.947063495977e-01 true resid norm > 1.792219139479e-03 ||r(i)||/||b|| 2.673762953736e-40 > 13 KSP preconditioned resid norm 1.638992222627e-01 true resid norm > 9.582599930477e-04 ||r(i)||/||b|| 1.429602001797e-40 > 14 KSP preconditioned resid norm 1.619641126005e-01 true resid norm > 5.129340629231e-04 ||r(i)||/||b|| 7.652323674838e-41 > 15 KSP preconditioned resid norm 6.298555854405e-02 true resid norm > 2.630662976191e-04 ||r(i)||/||b|| 3.924614493042e-41 > 16 KSP preconditioned resid norm 3.381480328186e-02 true resid norm > 1.405167435648e-04 ||r(i)||/||b|| 2.096331051529e-41 > 17 KSP preconditioned resid norm 2.188634069174e-02 true resid norm > 7.170524515175e-05 ||r(i)||/||b|| 1.069751035753e-41 > 18 KSP preconditioned resid norm 6.591386695704e-03 true resid norm > 3.789193478287e-05 ||r(i)||/||b|| 5.652994616344e-42 > 19 KSP preconditioned resid norm 5.988963507321e-03 true resid norm > 1.922304552896e-05 ||r(i)||/||b|| 2.867833841361e-42 > 20 KSP preconditioned resid norm 1.890309986311e-03 true resid norm > 9.615814496304e-06 ||r(i)||/||b|| 1.434557192470e-42 > 21 KSP preconditioned resid norm 1.322603958495e-03 true resid norm > 4.933661540922e-06 ||r(i)||/||b|| 7.360395368965e-43 > 22 KSP preconditioned resid norm 6.147516465031e-04 true resid norm > 2.434568419681e-06 ||r(i)||/||b|| 3.632066361467e-43 > 23 KSP preconditioned resid norm 2.443285908381e-04 true resid norm > 1.258673156233e-06 ||r(i)||/||b|| 1.877780223336e-43 > 24 KSP preconditioned resid norm 1.903790820937e-04 true resid norm > 6.814992942512e-07 ||r(i)||/||b|| 1.016710248109e-43 > 25 KSP preconditioned resid norm 5.897463754127e-05 true resid norm > 3.612787156069e-07 ||r(i)||/||b|| 5.389818825638e-44 > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-10, divergence=1e+300 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: bjacobi > number of blocks = 1 > Local solve is same for all blocks, in the following KSP and PC > objects: > KSP Object: (sub_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (sub_) 1 MPI processes > type: ilu > out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > package used to perform factorization: petsc > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > > > MINRES with bjacobi: > > 0 KSP preconditioned resid norm 1.400513201683e+03 true resid norm > 7.911033287257e+31 ||r(i)||/||b|| 1.181672927444e-05 > 1 KSP preconditioned resid norm 2.320798481784e-13 true resid norm > 5.094603145770e-03 ||r(i)||/||b|| 7.609820860095e-40 > KSP Object: 1 MPI processes > type: minres > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-10, absolute=1e-10, divergence=1e+300 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: bjacobi > number of blocks = 1 > Local solve is same for all blocks, in the following KSP and PC > objects: > KSP Object: (sub_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (sub_) 1 MPI processes > type: ilu > out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > package used to perform factorization: petsc > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=408848, cols=408848 > total: nonzeros=5918082, allocated nonzeros=5918082 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > > > On Wed, Jul 31, 2019 at 3:19 AM Jed Brown wrote: > >> Are you saying that the MINRES error is larger than CG error? In which >> norm? And which norm are you using for CG? (Output from >> -ksp_monitor_true_residual -ksp_view would be useful.) >> >> CG does find a solution that is optimal in a different inner product, >> though this is usually pretty benign unless you have a poor >> preconditioner. >> >> Mohammad Gohardoust via petsc-users writes: >> >> > Hi, >> > >> > I am trying to evaluate different solver/preconditioner combinations for >> > solving Richards' equation in porous media. The coefficient matrix is >> > symmetric positive definite and is usually solved by CG solver. I tried >> > MINRES with various preconditioners. The solution is acceptable but with >> > somehow considerable errors (compared to other solvers) which requires >> me >> > to decrease the tolerance threshold in my Picard iteration leading to >> > larger computation time. I would like to know if this is common for >> MINRES >> > and how to possibly overcome this? >> > >> > Thanks, >> > Mohammad >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: